some SOA facts

For some light humor, see: http://soafacts.com

Practical Guide for Federal Service Oriented Architecture (PGFSOA)

The American Council for Technology (ACT) / Industry Advisory Council (IAC) Enterprise Architecture (EA) Special Interest Group (SIG) (referred to as ACT/IAC EA SIG) just completed an internal review of the "Practical Guide for Federal Service Oriented Architecture (PGFSOA)", part of an IAC initiative. This document will be a product of the Federal CIO Council’s Architecture and Infrastructure Committee and the ACT/IAC's EA SIG and will provide guidance for federal agencies in implementing service oriented architecture. The draft was recently released for general comments and now approximately 300 comments are being resolved. The next version for broad comment is expected to be out in mid January. I was not able to gain access to the document itself so I volunteered my name for broad comment in January. About 35 persons have been actively supporting the writing. The objectives for the guide is to follow in the footsteps of "A Practical Guide to Federal EA" and build upon the Service-Component Based Architecture white papers that have been published. The collective group had wanted to rapidly create a document not too unlike the "Practical Guide to FEA", which provided practical guidance in support of agencies’ efforts to adopt SOA into their business, IT and EA practices.

Renee

some short notes from PegaWorld 2007

I am going to be brief on my notes from the conference as I will share all of the presentations with the team once they are available. Some assorted notes I captured are listed below.

Processes are the muscles; business rules are the nerves (quote by Ron Ross, Co-Founder and Principal of Business Rule Solutions, LLC, recognized internationally as being the "father of business rules”).

Applications are getting smaller. Different applications can share same rules and processes, bringing increased ability to rapidly respond to change.

Of the 20+ widely recognized workflow patterns (see http://www.workflowpatterns.com/vendors/documentation/BPMN-pat.pdf ), the patterns used 80% of the time are Sequential, Parallel Split and Join, and Decision Split (remember 80/20 rule?).

PowerPoint most widely used BPM tool, followed by Visio.

Business Activity Monitoring (BPM) is based on current (active) data; Business Intelligence (BI) is based on historical (and aggregated) data; Business Process Analytical (BPA) data is based on historical data for the intent of determining the future via statistical tools and predictive models. Examples of BI include Business Objects and Cognos. Examples of BPA include Hyperion, SAS and SPSS. BI is largely based on data warehousing.

PMML = Predictive Modeling Markup Language; PMML is a mark up language for statistical and data mining models. It can be used/generated by Pega.

SmartBPM = BPM + BI + SOA

Quote: “There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things.” ~Niccolo Machiavelli

Our brain (medulla oblongata) is genetically programmed to avoid/resist change (theorem of neurochemistry)

We resist change. Just for fun, the next time you shower, notice where you start lathering. The next day try somewhere new.

Strive for Business and IT fusion – not alignment. Alignment suggests trying to line up 2 or more things that are in motion? Ever try to deck a boat on a moving dock?

It is easy to buy technology pieces; it is hard to implement methodology.

What is the difference between a rule and a policy? The Ten Commandments are a policy; Thou Shalt Not Kill is a rule.

BPM is not just BPR – focus is on real time (or nearer real time)

top four enterprise architecture frameworks

Yesterday I was reminded of an article I had passed on to my team members months ago and subsequently forgot. Its on msdn and is titled “A Comparison of the Top Four Enterprise Architecture Methodologies” and was written by Roger Sessions. Sessions is the CTO of ObjectWatch Inc. and author of six books and dozens of articles and white papers. He serves on the Board of Directors of the International Association of Software Architects (www.iasahome.org).

Sessions takes the reader through the history of Enterprise Architecture and compares The Zachman Framework for Enterprise Architectures, The Open Group Architectural Framework (TOGAF), The Federal Enterprise Architecture and The Gartner Methodology. The article is easy to read through and uses a understandable and simple case to examplify the points. Sessions main conclusion being that none of the presented Enterprise Architecture methodologies is really complete. Each has its strengths and weaknesses. And most important - as Sessions also points out - the organization cannot succeed using any of them without commitment from the highest level of the organization. That is very much in line with my experience! Sessions offers a comparison of the methodologies and recommend using the best parts from each of them in the areas, where the organization has the most urgent needs. Here are some of Sessions points:

“Zachman tells you how to categorize your artifacts. TOGAF gives you a process for creating them.”

“TOGAF merely describes how to generate an enterprise architecture, not necessarily how to generate a good enterprise architecture.”

“FEA is the most complete of all the methodologies discussed so far. It has both a comprehensive taxonomy, like Zachman, and an architectural process, like TOGAF. FEA can be viewed as either a methodology for creating an enterprise architecture or the result of applying that process to a particular enterprise—namely, the U.S. Government.”

“The best summation of the Gartner practice that I have heard is the following: Architecture is a verb, not a noun.”

so much to do

I have been quiet because my head has been buried in a proposal. The Federal Government decided to give us only two weeks to respond to its RFP which we have been chasing for two years. We did a lot of work in advance but unfortunately its contents did not match the RFP and pretty much everything had to be thrown out. Then I made a short trip to see my mom in beautiful Hilton Head Island, South Carolina.

I have so many conferences and training coming up.

Next week I will be attending a free 2-day training session on the Methodology for Business Transformation (MBT) approach employed by the Department of Interior (DOI) Enterprise Architecture (EA) program. Northrop Grumman has been the lead contractor on this program, which created the MBT, based on the Federal Enterprise Architecture (FEA). The Office of Management and Budget (OMB) and General Accounting Office (GAO) have consistently ranked this program as having the highest of all ratings among all Federal Enterprise Architecture programs.

At the end of this month (October) I fly to Orlando for 4 days for PegaWorld 2007. Then next month I go to San Francisco for Oracle Open World 2007 where I will get to listen to people such as Michael Dell and Larry Ellison speak. I am not a groupie but it should be interesting what these guys have to say. I have never heard either of them speak live although I did listen to an audio version of Michael Dell's book seven years ago.

I get back on November 16, just in time to fly on the 17th to Europe for 2 weeks. ;-D

And between now and the first of March I have to attend three courses I have already paid for at the Learning Tree. I bought a training passport, not really realizing how difficult it was going to be to find time to attend four different training classes. I have had to schedule and reschedule and reschedule and reschedule again. In March it expires so I no longer have room to negotiate.

I cannot wait to get to Europe. ;-D

on perseverance

I am feeling tired... This past weekend was Labor Day -- a 3 or 4 day holiday for most folks... I was supposed to go visit my mother. But we are sitting waiting for the federal govt to release an RFP and we expect to be given only about a week since the govt has to award the contract THIS MONTH due to their fiscal budget. So I worked all 3 days... And today is Wednesday and still no RFP. :-( Anyway, we have been having lively discussions about Enterprise Architecture and Service Oriented Architecture and their boundaries -- talking about such things as it is wise to plan a large SOA project without considering EA? (no of course its not wise but how much do you need to elaborate?) And is it practical to adopt a SOA methodology that requires you to consider (or lay the groundwork for) EA (no its not...there are many projects out there which are candidates for SOA but the organization doesn't want to hear anything about EA) and how much groundwork is enough and how much is too much? Anyway, one of the folks involved in this discussion sent out the following quote (he was angry because he felt unfairly criticized -- he was and he also took it too seriously--sometimes its best to just let things roll off your shoulder).

"It is not the critic who counts: not the man who points out how the strong man stumbles or where the doer of deeds could have done better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly, who errs and comes up short again and again, because there is no effort without error or shortcoming, but who knows the great enthusiasms, the great devotions, who spends himself for a worthy cause; who, at the best, knows, in the end, the triumph of high achievement, and who, at the worst, if he fails, at least he fails while daring greatly, so that his place shall never be with those cold and timid souls who knew neither victory nor defeat."


"Citizenship in a Republic,"
Speech at the Sorbonne, Paris, April 23, 1910

interesting article on wikipedia

I have been busy doing some major off-line writing but wanted to share an article that I thought was kind of interesting..

CIA, FBI computers used for Wikipedia edits
Randall Mikkelsen

August 17, 2007 (Reuters) People using CIA and FBI computers have edited entries in the online encyclopedia Wikipedia on topics including the Iraq war and the Guantanamo prison, according to a new tracing program.

The changes may violate Wikipedia's conflict-of-interest guidelines, a spokeswoman for the site said on Thursday.

The program, WikiScanner, was developed by Virgil Griffith of the Santa Fe Institute in New Mexico and posted this month on a Web site that was quickly overwhelmed with searches.

The program allows users to track the source of computers used to make changes to the popular Internet encyclopedia where anyone can submit and edit entries.

WikiScanner revealed that CIA computers were used to edit an entry on the U.S.-led invasion of Iraq in 2003. A graphic on casualties was edited to add that many figures were estimated and were not broken down by class.

Another entry on former CIA chief William Colby was edited by CIA computers to expand his career history and discuss the merits of a Vietnam War rural pacification program that he headed.

Aerial and satellite images of the U.S. prison for terrorism suspects at Guantanamo Bay, Cuba, were removed using a computer traced to the FBI, WikiScanner showed.

CIA spokesman George Little said he could not confirm whether CIA computers were used in the changes, adding that "the agency always expects its computer systems to be used responsibly."

The FBI did not have an immediate response.

Computers at numerous other organizations and companies were found to have been involved in editing articles related to them.

Griffith said he developed WikiScanner "to create minor public relations disasters for companies and organizations I dislike (and) to see what 'interesting organizations' (which I am neutral towards) are up to."

It was not known whether changes were made by an official representative of an agency or company, Griffith said, but it was certain the change was made by someone with access to the organization's network.

It violates Wikipedia's neutrality guidelines for a person with close ties to an issue to contribute to an entry about it, said spokeswoman Sandy Ordonez of the Wikimedia Foundation, Wikipedia's parent organization.

However, she said, "Wikipedia is self-correcting," meaning misleading entries can be quickly revised by another editor. She said Wikimedia welcomed the WikiScanner.

trip report describing two recent Gartner conferences

From June 10 to June 15, I attended two consecutive Gartner conferences in Nashville. In the paragraphs that follow I attempt to share some of my observances in the areas of (a) application architecture, development and integration, and (b) enterprise architecture.

The contents below can be grouped into the following topics:
• General overview
• Web Oriented Architecture
• Web 2.0
• Web 3.0 and Web 4.0
• Random Observations
• Some SOA Best Practices
• Websites worthy of review
• Various statistics relevant to IT

General overview

The conference was interesting, not least of which Gartner predicts that SOA is already becoming legacy and among many other topics there is a new buzzword (since 2006, I believe) which is WOA (Web Oriented Architecture). One thing I did get some clarity on is that there is no such thing as a SOA architect! This may be obvious to you, the reader, but in the past year I have repeatedly heard this label used almost as a job position. SOA is simply a design pattern! The correct terminology is to speak of a solutions architect who employs among other design patterns, the SOA design pattern. ESB is also simply a design pattern. There was some discussion that ESBs may impede progress within the SOA strategy and it was forecasted that the ESB may even start to become obsolete within the next two years. Speaking of obsolescence, some analysts are actually even starting to consider AJAX as already becoming obsolete (although it will still be THE major technology for rich internet applications for the next couple of years). I was alerted to the importance in learning TOGAF in addition to FEA and DODAF.

Web Oriented Architecture

Web-oriented architecture (WOA) is based on the principles of REST. Representational state transfer (REST) is the style of the Web, and its principles should be followed for any SOA project.
What is the architecture of the Web, and how is it different from SOA? Although there are many formal definitions of SOA, the market is coalescing around a practical meaning for the term, which includes the following: SOA is about applications; while service orientation is a concept that can be applied to many things (storage area network is a service-oriented approach to storage), which does not make it SOA. SOA is about how one designs applications as the providers and consumers of a set of defined, agreed-upon interfaces (services). In this way, services are a unit of modularity in the same way that subroutines, objects and components are units of modularity. Modularity is how we have improved the ability of programs and our ability to create more complex and integrated software. A key concept in the current modularity that SOA provides is that it incorporates encapsulation. In other words, the service interface is the only means of accessing anything about that service or the information it manages. A key attribute of these new modules is that they are shared, not reused. This means that the definitions that constrain their operation are shared as well. Another thing that is critical to the value of SOA is the fact that the only thing these modules must have in common is the network. That network — a Simple Object Access Protocol (SOAP) network or its logical equivalent — is the only means of access. This removes any dependency on development time components. The definition of a service is not its WSDL or its implementation code, but how it is manifested over the wire, which eliminates any technological dependency on anything except the application wire protocol. The final assumption is that modules are mostly the same as a subroutine or other structured code component. Thus, some say that current SOA initiatives follow a different architectural pattern than the Web and, therefore, cannot be expected to deliver some of the benefits that are inherent in the Web-oriented pattern.
The only real difference between traditional SOA and the concept of WOA is that WOA advocates REST, an increasingly popular, powerful, and simple method of leveraging HTTP as a Web service in its own right (and carefully devised by the co-creator of HTTP, Roy Fielding.)
WOA itself is a reflection of the classic Reach vs. Rich argument, in that richness is an outcome of robustness and complexity but cuts down how much reach you have to others, particularly in heterogeneous communities. Some say that the modern SOA technologies and architectures have become a near-morass of complex standards, protocols, and products, and that the result is that the present list of WS-* standards is nearly incomprehensible.
This is making some people fall back to simpler and more straightforward methods that work, while hiding the protocol complexity in the application state. Some say that one doesn’t usually need anything more complex than XML and HTTP, one of the most scalable, proven, and widespread protocols on the planet, along with HTTP’s verbs: GET, POST, PUT, DELETE.
REST does not handle enterprise issues like two-phase commit, messaging, and asynchronicity. WOA is definitely not the hammer for every job. Certain applications, particularly in the high-end of the enterprise, require the more sophisticated portions of the SOA stack (don’t forget, WOA is a subset of SOA and is not disjoint as long as you follow the principles of service orientation). However, for many uses, WOA is the most interoperable, easiest to implement, and most highly scalable technique for building open web services.

Web 2.0

Much of the conference was centered on Web 2.0. One of the languages used in AJAX web application programming is "JavaScript Object Notation" (JSON) which is pronounced like the English name Jason. JSON is based on a subset of the JavaScript Programming Language. JSON is a fast, compact representational format that has advantages when used with JavaScript (as is found in browsers). XML, when used with browsers, must be parsed and processed by JavaScript while JSON is simply "consumed" directly. This lightweight data-interchange format is easy for humans to read and write, and easy for machines to parse and generate. JSON is a text format that is completely language independent but uses conventions familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON an ideal data-interchange language. It provides a simple alternative to XML for asynchronously transmitting structured information between a client and its server. In December 2005, Yahoo! began offering some of its Web Services optionally in JSON. Google started offering JSON feeds for its GData web protocol in December 2006.

Web 3.0 and Web 4.0

Have you ever wondered what the Internet will look like in the future? Most people have heard of some aspect of what has been called web 2.0. You know things like social networking, lightweight programming, web as a platform, and collective intelligence. Most people recognize these by their street names Facebook, AJAX, Google Maps, and wikipedia respectively.
Many people refer to Web 3.0 as the semantic web. The semantic web is where machines read web pages as humans do today. A web where search engines and software agents peruse "the Net" and find us what we're looking for. Some of the technologies required to perform such tasks exist today including the Resource description Framework (RDF) and the Web Ontology Language (OWL). These work by adding all sorts of machine readable metadata to human readable web pages.
The web 2.0 technologies are beginning to trickle into real-world sites, service experiences, and tools. Examples of these technologies can be seen with Yahoo!'s food site, which is driven by semantic metadata, the semantic portal developed by Radar Networks, the Jena development platform being creating at HP, and the Oracle Spatial database.
Just as much of life, the semantic web is a good-news, bad-news thing. You can do all these wonderfully complex queries, but it takes a tremendous amount of time and metadata to make it happen. This is due to having to completely re-annotate the entire Web in order to make existing web pages machine readable.
For this reason, many researchers opt for a very different approach to the semantic web. Instead of an overhaul of Web formats, they're building agents that can better understand the Web pages of today. The pages aren't easier to read, but rather the software agents are getting smarter.
There is yet another view of the semantic web taken by so called "semantic searchers". Rather than providing automatic information retrieval, semantic search engines seek to improve on the Google-like search model that we've grown so accustomed to. the idea is to move beyond mere keyword searches to a better understanding of natural-language queries. This type of natural-language processing has been in development for years and now is beginning to find its way onto the public Web.
One could argue, however, that while many associate Web 3.0 and the Semantic Web, the two are far from synonymous. There are countless other concepts ready to alter our online experiences of the future. Many of these concepts go beyond semantics using space, images, and sound.
One such concept is the so called 3D Web, a Web you can walk through. This has been seen as an extension to the "virtual worlds" popping up on today's Web. It has been said that the Web of the future will be the big alternate universe reminiscent of Second Like and There.com.
As with everything in like there are those who scoff at this notion of the 3d Web. They see the 3D Web as a re-creation of our existing world. On the 3D Web, you can take a virtual tour of an unfamiliar neighborhood and shop for houses or visit famous sites you've never seen. Google earth already offers a similar experience. The problem is that 3D only goes so far and doesn't enhance the 2D world of text, pictures, and video. An interesting idea beginning to emerge is that of a media-centric Web which offers not only language-based search but pure media search. Today's searching mechanisms rely primarily on keywords, even when searching for images, videos, and songs, this is a woefully inadequate approach. Some companies are looking to reinvent media search, hinting at a world where we search for media with other media, not just keywords.
Speaking of web 2.0 and web 3.0, some are already talking about a web 4.0, dubbed the ubiquitous (or pervasive) web. The Web already reaches beyond the desktop, to cell phones and handhelds, but who says it has to stop there? What about having the windows in your home open automatically when the temperature changes? By leveraging mesh networks, wireless networks consisting of tiny nodes that can route data to and from almost anywhere, the possibilities are endless. However, regarding web 2.0, web 3.0, and web 4.0, perhaps we should all avoid getting into a naming competition.

Random Observations

Ruby is an open-source all-purpose, object-oriented scripting language that was developed in Japan in 1995. Rails is a open-source, full-stack, Model-View-Controller (MVC) web framework, designed using the Ruby programming language. Because of Ruby's dynamic nature, Rails provides many exciting features for programmers, such as easy database access and methods to utilize some of the newest technologies on the web.
Service Component Architecture (SCA) is another significant new development. Many of the vendors in the consortium that created it (www.osoa.org) have products in the pipeline that conform to SCA. However, because of the breadth of capability, and the sheer number of vendors involved, Gartner expects the formal ratification process to take a long time. They also expect that this will suffer from the usual challenges with vendor-led standards, in that many vendors will create implementations that conform to the standard, but that also contain significant extensions that are useful to customers. This will leave organizations with the choice between retaining interoperability, and conforming to the standard at the cost of additional programming time or by using these convenience features and accepting limitations on the interoperability.
Start to look for higher-level standards, such as WS-SX and WS-RX, within the near term, as they become mature (WS-SX = Web-Services Secure Exchange; WS-RX = Web-Services Reliable Exchange).
“Net Gen” kids now entering workforce and consumer market and will demand web 2.0 capabilities, potentially having a large impact on corporations. There are 80 million people ages 8 to 28 in U.S. alone.
SOAs don’t finish; it’s a journey, not a destination.
Three main ways to approach SOA: REST/POX, web services, and proprietary methods.
English has been dominant language in blogs but Japanese is about to catch up or pass English as dominant language.
Blogs can be particularly helpful for technology evangelism, project management, standards work, technology change management. Social networking is useful in finding needed expertise (i.e. LinkedIn, myspace, facebook). For instance, one could use facebook or myspace in a corporation to describe specific projects, technologies, skills, etc. One idea is to make a matrix of stakeholders vertically and horizontally and brainstorm on potential reasons why such people might collaborate.
Mashups are predicted to become dominant form of composite applications. These do not have to be limited to map data. A mashup is simply the idea of bringing together two or more different data sets.
AJAX is not new. Pioneers in AJAX technology include outlook web access (circa 1998) and desktop.com (circa 1999).
BPM really has very little to do with IT; its mostly a management discipline.
TOGAF is the defacto enterprise architecture framework standard. TOGAF provides specific methods in its ADM. TOGAF is frequently applied to SOA.
WS-I provides ws-i conformance tools. Can be downloaded from WS-I site. Mindreef, IBM, Microsoft, Parasoft all have versions of these tests.
Aspect-oriented programming (AOP), and aspect-oriented software development (AOSD) are gaining momentum. See wikipedia for more information.
In late 2006, Target was sued by National Federation for the Blind for having a web site inaccessible to the blind. Results could have large impact.
Microsoft introduced Silverlight about 2 months ago to compete against Adobe’s Flash.
About two weeks ago Microsoft released a tool to allow non-IT people to build mash-ups.
Mission critical applications deployed as a mashup is risky as its dependent on external data.
There are 100+ AJAX vendors, including tools and vendors Adobe Flash, Eclipse RCP, Microsoft WPF and Silverlight, and IBM Expeditor.
Sun has introduced JavaFX to compete with Flash and Silverlight.
There appears to be no correlation between EA maturity and how long one has been doing it. Architecture impact remains low. This indicates that there a lot of companies are doing lots of work but it is not making any difference.
Try to avoid modelitis (the compulsive drawing of lots of pretty models) even though it is fun to play with cool tools.
Avoid “management by magazine”.
Know standard ISO/IEC 42010 or ANSI/IEEE 1471-2000
DoDAF, FEAF, TOGAF, Gartner, Zachman are all examples of architecture frameworks.

Some SOA Best Practices

Start by looking at REST/POX. If a problem can be solved via REST/POX, then focus there. If the problem needs more capabilities than REST/POX provides, then move up to Web services, and eventually to proprietary approaches, as necessary. This may seem to increase the complexity of a single solution, but in reality, it simplifies the overall solution. Many features will be implemented with the simplistic REST/POX approach, which will reduce the number of complex proprietary approaches and more-complicated Web services approaches. Use the right tool for the right job, and ensure that your toolkit includes REST/POX, Web services and proprietary approaches.
Today's programmatic access uses a combination of REST/POX, Web services and proprietary approaches. For new development, start simple and step up as required.
Create a service when:
• Crossing a boundary
• Seeking more than one reuse scenario
• Providing a significant business process
• Invoking a more-complex process
• Requesting information
• Exposing a high-level API
• Exposing public data
• Needs to be shared
• Needs to be explicitly defined and described
• Changing frequently
• Describing a discrete function
• You want capabilities instead of features
Avoid a service when:
• Performance is paramount
• Working within one system
• Transferring large data sets
• Executing transactions
• Data is of extremely fine granularity

Websites worthy of review

http://www.writely.com
http://www.riya.com
http://iscrybe.com/main/index.php
https://www.youos.com
http://www.zoho.com
http://www.pageflakes.com
http://googlesuggest.com
http://tiddlywiki.com
http://secondlife.com
http://www.socialtext.com
http://www.goowy.com
http://www.digg.com

Various statistics relevant to IT

• Gartner predicts that SOA will be partly used in more than 80% of new, mission-critical applications and business processes designed by 2010.
• Through 2009, organizations that do not consider data integration and data quality as critical components of SOA will double their risk of failure in SOA-related projects (0.8 probability).
• Through 2008, the majority of Global 1000 companies will adopt the technology-related aspects of the Web 2.0 phenomenon, but fail to adopt the aspects of Web 2.0 that have a social dimension. The result will have a minimal business impact (0.6 probability).
• Organizations that align their BPM and SOA initiatives in 2007 will double their likelihood of becoming industry leaders by 2011 (0.7 probability).
• Web architectural style will be used in more than 75% of new applications by 2012 (0.8 probability).
• By 2012, 75% of practitioners of the current style of SOA will have failed to measurably improve their application agility (0.8 probability).
• By 2012, current forms of service-oriented architecture (SOA) will be considered legacy architectural styles (0.9 probability).
• By 2010, Web mashups will be the dominant model (80%) for the creation of composite enterprise applications (0.7 probability).
• The high cost and complexity of information integration continues to eat at 30% to 40% of most corporate IT budgets.
• Organizations building an enterprise architecture will achieve marginal to no benefits during the first three years following its implementation without explicit links to software process and behavioral change (0.8 probability).
• By 2010, 40% of existing EA programs will be stopped (0.9 probability). Failure to define value leads to a failure to deliver value, which leads to a failure of confidence, which leads to EA program failures.
• Fully mature semantic reconciliation tools will not be available until 2011 (0.7 probability). By the end of 2009, 40% of a global company's data will be defined in some way by XML (0.7 probability).
• By the end of 2009, 75% of the Fortune 500's interapplication messaging infrastructure will be formatted in XML (0.7 probability).
• By 2010, web-oriented architecture will account for 60% of SOA development in the enterprise (0.7 probability).
• Through 2010, lack of working SOA governance arrangements will be the most common reason for SOA failure (0.8 probability).
• SOA is currently at trough of disillusionment. 2-5 years mainstream adoption.
• By 2010, about $1 of every $3 spent on IT professional services will be spent on a solution using SOA, Web services or Web 2.0 (0.8 probability).
• According to Gartner Dataquest, in 2006, the worldwide market for IT professional services involving Web services or Web 2.0 was $58.9 billion, and the market will grow at a compound annual rate of 27% to reach $152.4 billion by 2010. Growth will be fastest in development and integration services through 2009 pending the availability of a critical mass of commercially available SOBAs.
• Through 2009, lack of SOA, Web services and Web 2.0 skills will require enterprises to obtain expertise from service or software providers for 70% of their projects involving these technologies (0.8 probability).
• Through 2010, fewer than 25% of large companies will have developed the technical and organizational skills needed to deliver enterprisewide SOA (0.8 probability).
• By 2008, 50% of companies will use Web 2.0-style services in their SOA initiatives (0.8 probability).
• Gartner predicts that SOA will be partly used in more than 80% of new, mission-critical applications and business processes designed by 2010.
• By 2011, at least 80% of all commercial software solutions will include elements of open source (0.8 probability).
• By 2010, WOA will account for 60% of SOA deployment in the enterprise (0.7 probability).

This is a co-worker's brother-in-law (see article below). He is all over the place, including at last week's Gartner's SOA conference that I attended. I will be writing a trip report as time permits although perhaps not as lengthy as my last trip report as time is becoming a bit of a luxury. The conference was very interesting, not least of which they predict that SOA is already becoming legacy and among many other topics there is a new buzzword (since 2006, I believe) which is WOA (Web Oriented Architecture). WOA basically refers to the REST method. Gartner indicated that best practice is to adopt WOA when you can instead of SOA. One thing I did get some clarity on is that there is no such thing as a SOA architect! This may be obvious to all of you but in the past year I have repeatedly heard this label used almost as a job position. SOA is simply just a design pattern! The correct terminology is to speak of a solutions architect who employs among other design patterns, the SOA design pattern. We all know that the ESB is also just a design pattern. And now there is discussion that ESBs may actually impede progress within the SOA strategy and it is forecasted that the ESB may also become obsolete within the next two years. Speaking of obsolescence, did you know that some actually even consider AJAX as already becoming obsolete (although it will still be THE major technology for rich internet applications for the next couple of years). Anyway, I have a lot to report -- it was a very productive week as I attended two separate conferences, one on SOA and one on Enterprise Architecture (hint: we need to all learn TOGAF in addition to FEA and DODAF) -- and I will try to provide a summary of some of what I have learned by the end of the week. I have all of the PDFs if anyone is interested. I also ordered the multimedia CD from the SOA conference (not the EA one though as I thought this was all just too much money). By the way, Java and C# are out; Ruby on Rails is now the *hot* language and web application foundation with Python being a close second hot language. (I wish things would slow down a bit so I could get a good grasp *smile*) And hey if anyone can figure out what to do with Second Life, please let me know! I know IBM hosts conferences there as well as a multitude of other companies but I find it very difficult to use.

How Much Will Your SOA Cost?
By: David S. Linthicum, CEO, Linthicum Group
Tuesday, March 20, 2007

I'm consulting now...at the project and strategy levels...and thus finding that a lot of real work needs to be done to get SOAs up-and-running. For most organizations, the first step of their SOA project is to figure out how much this SOA will cost. Thus, you can budget appropriately and obtain the funding.
It’s a good first step, but most organizations that want to build an SOA don't have a clue how to approach the cost estimation process. In many cases, they grossly underestimate the cost of their SOA, hoping their bosses and accountants won't notice later. In other words, go in low to get the approval, and reveal the higher costs later after it's too late...the investment has been made. Not a good management practice, if you ask me, but a pretty common one.
So, how do you calculate the cost of an SOA? While you can't cost out an SOA like a construction project, many of the same notions apply, including: understanding the domain, understanding how much required resources cost, and understanding how the work will get done. Moreover, understand what can go wrong and account for it.
Here are some very “general" guidelines:
Budget to budget. The problem that I'm seeing over and over again is that cost estimates are provided without a clear understanding of the work needed to be done. Indeed, you need to budget some time to create the budget. This means understanding the domain in detail, including:
1. Number of data elements
2. Complexity of data storage technology
3. System complexity
4. Service complexity
5. Process complexity
6. New services needed.
7. Enabling technology
8. Applicable standards
9. Potential risks
Typically it's expressed as:
Cost of SOA = (Cost of Data Complexity + Cost of Service Complexity + Cost of Process Complexity + Enabling Technology Solution)
For instance:
Cost of Data Complexity = (((Number of Data Elements) * Complexity of the Data Storage Technology) * Labor Units))
• Number of Data Elements being the number of semantics you're tracking in your domain, new or derived.
• Complexity of the Data Storage Technology, expressed as a percentage between 0 and 1 (0% to 100%). For instance, Relational is a .3, Object-Oriented is a .6, and ISAM is a .8.
So, at $100 a labor unit, or the amount of money it takes to understand and refine one data element, we could have:
Cost of Data Complexity = (((3,000) * .8) * $100)
Or, Cost of Data Complexity = $150,000 USD Or, the amount of money needed to both understand and refine the data so it fits into your SOA, which is a small part of the overall project by the way.
If you get this, you can get the rest of the cost analysis procedure; just reapply the same notions to:
Cost of Service Complexity
Cost of Process Complexity
Enabling Technology Solution
Some things to remember:
1. This is not really metrics (e.g., function points), because we really don't have historical data to abstract here. In essence, you need to use your own project management and project costing methods; just apply them to this new approach, using the formulas I'm suggesting above.
2. Count on 10 to 20 percent variations in cost for the simple reason that we've not walked down this road before. As we move from project to project, we'll get better at costing out SOA.
3. Make sure you dial in at least 2 major mistakes, meaning, selecting the wrong vendor, or hiring the wrong architect. You may encounter more, but it will almost never be less.
4. Make sure to change cost estimates as scope creep occurs, and it always does. The nice things about using formulas such as the ones I’m expressing here is that, as change occurs, you can quickly see the effect on the budget. Moreover, as change occurs later in the SOA projects, the cost of change goes up exponentially.
Finally, here is a sensible, no-nonsense approach to costing out SOA. While the actual numeric assumptions may be debatable, it's the approach that's refreshing. It would be great to see people start using this model, sharing any data points and providing feedback so it could be refined. Clearly, this would benefit the IT community a lot. I plan to develop a much more sophisticated model in the near future. If you’re interested, let me know, I’ll post it on my Web site.

David Linthicum (Dave) is an internationally known application integration and service oriented architecture expert. In his career Dave has formed many of the ideas behind modern distributed computing including EAI (enterprise application integration), B2B application integration, and service oriented architecture (SOA), approaches and technologies in wide use today.

Currently, Dave is the CEO of the Linthicum Group, LLC, a consulting organization dedicated to excellence in SOA product development, SOA implementation, and corporate SOA strategy. Dave is formerly the CEO of BRIDGEWERX, and is also the former CTO of Mercator Software and has held key technology management roles with a number of organizations including CTO of SAGA Software, Mobil Oil, EDS, AT&T, and Ernst and Young.

Blogging reaching its peak In 2007

Blogging may witness its peak time in the year 2007, with the number of blogs leveling out at 100 million. According to technology predictions by analysts at the research company Gartner, almost 200 million people have already stopped writing their blogs. Daryl Plummer, chief Gartner fellow, said that although some people who make a blog get bored and forget about maintaining it, others love it and are committed to keeping it up. "A lot of people have been in and out of this thing," Plummer said, according to BBC. "Everyone thinks they have something to say, until they're put on stage and asked to say it." Last month the blog-tracking firm Technorati reported that nearly 100,000 new blogs were being created every day, and 1.3 million blog posts were written. The firm, which is keeping track of more than 57 million blogs, says around 55% are "active" and updated at least every three months.

ROI and SOA

This video was also published by TIBCO. Good to see someone has a sense of humor. *smile*

SOA

I just spent a week at two Gartner conferences, which I will blog about later. But I was shown this cute little video:

back from vacation/study on software piracy

I am back from a wonderful vacation to the Black Hills! Anyway, I thought I would share this information concerning software piracy.

From IDC

Worldwide Software Piracy Rate Holds Steady at 35%, Global Losses Up 15%

15 May 2007

Piracy Rate in China Drops 10% in Three Years

WASHINGTON, D.C., May 15, 2007 — A new study reveals that 35% of the software installed in 2006 on personal computers (PCs) worldwide was obtained illegally, amounting to nearly $40 billion in global losses due to software piracy. Progress was seen in a number of emerging markets, most notably in China, where the piracy rate dropped ten percentage points in three years, and in Russia, where piracy fell seven percentage points over three years.

These are among the findings of the fourth annual global PC software piracy study released today by the Business Software Alliance (BSA), an international association representing the commercial software industry. The study was conducted independently by IDC, the information technology (IT) industry's leading global market research and forecasting firm.

"The good news is we are making progress, however, we still have a lot of work to do to reduce unacceptable levels of piracy," said BSA President and CEO Robert Holleyman. "These significant losses translate into negative impacts on IT industry employment, revenues, and financial resources available for future innovation and the development of new technologies."

Worldwide, for every two dollars of software purchased legitimately, one dollar was obtained illegally. Global losses increased in 2006 by more than $5 billion (15%) over the previous year. Of the 102 countries covered in this year's study, piracy rates dropped moderately in sixty-two countries, while increasing in thirteen.

China's piracy rate dropped four percentage points for the second consecutive year and has dropped ten percentage points in the last three years, from 92% in 2003 to 82% in 2006. By reducing China's piracy rate by ten percentage points over three years, $864 million in losses was saved, according to IDC. The reduction in the piracy rate and the savings are the result of government efforts to increase the use of legitimate software within its own departments, vendor arrangements with PC suppliers to use legitimate software, and increasing industry and government education and enforcement efforts. The legitimate software market in China grew to nearly $1.2 billion in 2006, an increase of 88% over 2005. Since 2003, the legitimate software market in China has grown over 358%.

"Considering the vast PC growth taking place in the Chinese IT market, this continued decline in China's software piracy rate is quite promising," said Holleyman. "BSA is encouraged by the commitment from the Chinese government to ensure legal software use. We look forward to continued dialogue between the US and China aimed at addressing issues that affect both economies."

And in Russia, the piracy rate decreased by seven percentage points since 2003, down from 87% in 2003 to 80% in 2006.

In addition, the study indicates that even relatively low piracy rates can amount to huge losses in large markets. For example, while the U.S. had the lowest piracy rate of all countries studied at 21%, it also had the greatest total losses at $7.3 billion. China saw the second highest losses at $5.4 billion with a piracy rate of 82%, followed by France with losses of $2.7 billion and a piracy rate of 45%.

Other key findings:

In more than half of the 102 countries studied, the piracy rate exceeded 60%. In approximately one third of the countries, the piracy rate exceeded 75%.
Emerging markets in Asia/Pacific, Latin America, Eastern Europe, and the Middle East and Africa accounted for one-third of PC shipments, but only 10% of spending on PC software.
The European Union (EU) and Canada continue to have high losses despite low piracy rates. The EU had losses of $11 billion with a 36% piracy rate, while Canada had losses of $784 million with a 34% piracy rate.
Over the next four years, businesses and consumers worldwide will spend $350 billion on PC software. If current trends continue, the study predicts more than $180 billion worth of PC software will be pirated during that period.

"A number of factors contribute to regional differences in piracy: the strength of intellectual property protection, the availability of pirated software, and cultural differences," said John Gantz, chief research officer at IDC. "Reducing software piracy around the world will take much more work and investment, but those efforts will pay off in the form of stronger local IT industries that drive broader economic growth."

"The critical elements of the global fight against software piracy are education, strong government policy, and enforcement," said Holleyman. "Expanded access to the Internet in emerging markets is making piracy more convenient, which makes it all the more imperative to keep up anti-piracy efforts."

I saw my first Mac!

Well, after having worked in IT for some 25 years, I finally broke down and went to see what a Macintosh computer looked like. What I like about it is that it runs on top of a variant of a Unix-based OS. What I don't like about it is its UI. I know that is one of the selling features of the Mac. I just didn't care for how on its current OS version all of the icons are on the bottom and if you cursor over them they expand in size and kind of pop out at you. Personally, I just find this disorienting. Just like I also detest flashing HTML. What I like about it is that hackers write viruses for Windows since Mac is only about 5.8 percent of the personal computer market. Many people don't have load anti-virus software on their Mac. This is from a report from Macworld, "According to IDC’s report, the growth puts Apple’s market share at 5.8 percent (fourth place overall), ahead of Toshiba at 4.2 percent. Dell topped the U.S. market with 31 percent, but suffered a negative growth rate of -6.7 percent. The top 5 is rounded out by HP with a 22 percent share and Gateway with a 6 percent share." HP is currently the dominant pc maker. I am writing on a HP right now.

a SOA class at IBM

I attended a free 3-day class (free for IBM partners) titled “Designing SOA Solutions with the IBM Foundation” from April 30 to May 2, 2007. I found the class very useful. Although the class somewhat deviated from the planned agenda, I found the dialog quite intriguing. All of the participants had SOA experience and knowledge and came with plenty of questions and offered valuable feedback and real life experiences. Incidentally, out of 10 people, I was the only female (typical) and only one of two Americans. The other American was also from my company. I was able to get a quick glance at some of the framework provided within IBM’s rather closely guarded and propreitary closed Service-Oriented Modeling and Architecture (SOMA) methodology. SOMA is IBM’s approach to service modeling, service design and service deployment. SOMA covers a broad scope and implements service-oriented analysis and design (SOAD) through the identification, specification and realization of services, components that realize those services (a.k.a. "service components"), and the flows that can be used to compose services. A description of SOMA is available in the article "Service-oriented Modeling and Architecture: How to Identify, Specify and Realize Services". SOMA includes an analysis and design method that extends traditional object-oriented and component-based analysis and design methods to include concerns relevant to and supporting SOA. IBM has integrated SOMA with its RUP methodology and its tools, particularly the Websphere line of products.

pre-conference on BPMN, XPDL, and BPEL

This is part of an e-mail I sent when asked about attending yet another conference on SOA and BPM. I said I was more interested in just one of their pre-conference sessions than the conference itself. The pre-conference is basically a tutorial on BPMN, XPDL, and BPEL.

BPMN (Business Process Modeling Notation) is easy enough to pick up. As long as you know good practices when it comes to modeling, the syntax is not that important. BPMN, for instance, will dictate that a process merge uses a certain diagram notation. Many of the BPM vendors are moving towards BPMN because it’s a standard just like a UML class diagram is a standard. You learn the standard and then you can immediately grasp what's represented in any diagram based on that standard. (Assuming of course that what is shown in the diagram makes sense--LOL--you can create a bad diagram in any tool.) Pegasystems Rules Process Commander (PRPC) is capable of using BPMN but they also have their own preferred notation. As far as memorizing the different shapes, I wouldn't bother. You can easily enough learn these if you work with the diagrams for a day or so.

The topics that are of more importance is XPDL and BPEL. Sure, I know what they do and what their purpose is. I wouldn't mind delving a little deeper into specifics of these standards. However, the truth is that a good BPM tool will create XPDL and BPEL and you can use a specialized editor to manipulate it. Kind of like one usually wouldn't type in HTML or XML (any more they wouldn't--there are too many tools and its good practice to stay at the highest level of abstraction possible and let tools do for you what they can) unless they had to tweak something very specific. And when you have to tweak something that specific, you probably need a reference book anyway. ;-) Still, it would be worthwhile to get in "under the hood" as Dr. Warner likes to say and better understand these two standards. I have looked at code samples and so forth -- at least for BPEL -- and would find value in attending this. The pre-conference is $295. This is still a little expensive for one day but conferences and workshops and training are always expensive. It is certainly cheaper than $2,495.

article today on the Myth of High-Tech Outsourcing

The Myth of High-Tech Outsourcing
By Catherine Holahan

April 24, 2007 9:57AM
There is so much global demand for employees proficient in programming languages, engineering, and other skills demanding higher level technology knowledge that outsourcing can't meet all U.S. needs. "There would have been a lot more than 147,000 jobs created here, but our companies are having difficulty finding Americans with the background," says William Archey, president and chief executive of the AeA.

High-tech employees are back in demand. The U.S. technology industry added almost 150,000 jobs in 2006, according to an Apr. 24 report by the American Electronics Assn. (AeA), an industry trade group. That was the largest gain since 2001, before the implosion of the tech bubble resulted in the loss of more than 1 million jobs in three years.

The findings counter concerns -- sometimes voiced by opponents of outsourcing -- that high-tech jobs are being sent overseas.

There's plenty of domestic demand for a host of I.T. jobs, says Katherine Spencer Lee, executive director of Robert Half Technology, an I.T. staffing company headquartered in Menlo Park, Calif. On average, it is taking 56 days to fill full-time I.T. positions, she says. Firms that want I.T. managers are looking at an even longer search -- about 87 days. And the wait is only getting longer.

Employment Highs

Workers well-versed in the emerging Web, with its emphasis on user-generated content, are having little trouble landing jobs. "The big buzz right now is the whole Web 2.0 space," says Spencer Lee, adding that anyone with a background in operating systems or knowledge of .net or Asynchronous JavaScript and XML (Ajax) is especially sought after. "We have seen pretty big demand."

Unemployment for engineers, computer programmers, software developers, and other I.T. professionals is at the lowest rate in years. Less than 3% of computer systems designers are out of work and less than 2% of engineers are sitting at home searching the classifieds, according to the AeA study. U.S. unemployment across the board is about 5.1%. "I think this is a bit of a rebounding from the burst," says Karen Carruthers, director of marketing at Rostie & Associates, an I.T. staffing firm with offices in Boston, San Diego, and Toronto.

Outsourcing Slowdown

So what about all those jobs supposedly headed offshore? To be sure, companies have relocated call centers and even some software development jobs to places such as Bangalore, India, Prague, and Russia, where some labor costs are lower and skilled workers abound.

But there is so much global demand for employees proficient in programming languages, engineering, and other skills demanding higher level technology knowledge that outsourcing can't meet all U.S. needs. "There would have been a lot more than 147,000 jobs created here, but our companies are having difficulty finding Americans with the background," says William Archey, president and chief executive of the AeA.
One culprit is the dearth of U.S. engineering and computer science college graduates. Second, immigration caps have made it difficult for highly skilled foreign-born employees to obtain work visas. Congress has been debating whether to increase the numbers of foreign skilled workers allowed into the country under the H-1B visa program.

Marketing to Students

American universities and high schools are trying to fix the first problem by encouraging more students to get involved in math and science careers. The percentage of college freshmen planning to major in computer science dropped 70% between 2000 and 2005--the same years the tech sector declined spectacularly. Schools and companies are trying to counteract this with programs that teach the practical application of tech skills.

Money can help fuel interest. So, certainly the average high-tech salary of $75,500 in 2005, compared with the average private-sector wage of $40,500, should gradually encourage more Americans to seek bachelor of science degrees. Archey believes there must also be a cultural shift in how Americans see high-tech jobs: "Kids think it must be pretty boring to go into high-tech because if you do, you're a geek," says Archey. "We have to do a much better job showing how exciting the world of technology is."

David Bair, national vice-president of technology recruiting at Kforce, says that the U.S. needs a marketing campaign around technology. "We are going to have to make sure that we have students coming into the space," says Bair. "We need to let people know this is an unbelievable career opportunity for individuals."

Foreign Legion

Then there's the option of letting more skilled foreign workers enter the U.S.--though it meets with opposition from lawmakers who view limits on work visas as a safeguard for highly skilled U.S. citizens.

Even if the restrictions are lifted, some skilled foreign workers may find plenty of reasons to stay abroad. Increasingly, there are opportunities for talented high-tech professionals in their home countries. "Ten years ago, if you had somebody really bright coming out of a European or Asian university, there was nowhere to go other than the U.S.," says Archey. "We no longer have a monopoly on that."

Some employees are finding they have a better quality of life working in their home countries, says Steve Van Natta, president of V2 Staffing, a consulting services company in Shelton, Conn., that specializes in software development. This is especially true for experienced employees who are familiar with the operations of U.S. companies--the kind most in demand domestically. "If anyone comes over to the U.S. to get experience, their stock gets even higher when they go back home," says Van Natta.

Training and ROI

To meet the demand, Van Natta and other recruiters say that companies will need to be more flexible with their requirements and train capable employees without extensive experience in the specific area of need. "You take a smart person who comes in and doesn't necessarily have your industry experience but is a good developer and give them the functional training that they need," says Van Natta.

And unlike a half-decade ago, demand is likely to remain for now, recruiters say. Many of the available jobs are for companies that have proven returns--not ideas that have yet to pan out. "People are hiring someone not just to do one task," says Robert Half Technology's Spencer Lee. "The hiring here is based on ROI [return on investment]."

20 key "MUST HAVE" features for any BPM tool

Rashid Khan, the man who started the company Ultimus and has written a book titled Business Process Management identifies 20 features that he feels are key to any BPM product:

• Robust business rules
• Role-based routing
• Relationship routing
• Relative routing
• Parallel routing
• Ad hoc routing
• Queues and groups
• Process rollback
• Support for sub-processes
• Escalations and exceptions
• Flexible forms support
• Web-based architecture
• Automation agents
• Custom views
• Simulation
• Process documentation
• Status monitoring
• Authentication and security
• Distributed user administration
• Task delegation and conferring

Likewise, Khan identifies ten key modules or capabilities essential to any complete BPM suite environment:

• Graphic designer
• Collaboration design
• Modeling
• Organization charts and directory integration
• Multiple client interfaces
• Business metrics and monitoring
• BPM Administrator
• Web services and integration
• Database connectivity and transaction processing
• Scalable BPM server

problem of application monitoring

Background

Applications are frequently degraded or otherwise inaccessible to users. It is hard to find out in any kind of timely manner that there have been application problems. Indeed, sometimes one does not learn that an application has been down until a significant amount of time has passed. This indicates a need for a tool or a set of tools that are able to indicate a high-level status of the applications. Such a tool falls within the set of tools collectively referred to as application monitoring. This paper presents a brief discussion on these methods, tools, and the justification for the adoption of such technology.

Discussion

Monitoring applications to detect and respond to problems - before an end user is even aware that a problem exists - is a common systems requirement. Most administrators understand the need for application monitoring. Infrastructure teams, in fact, typically monitor the basic health of application servers by keeping an eye on CPU utilization, throughput, memory usage and the like. However, there are many parts to an application server environment, and understanding which metrics to monitor for each of these pieces differentiates those environments that can effectively anticipate production problems from those that might get overwhelmed by them.

When applied in an appropriate context, application monitoring is more than just the data that shows how an application is performing technically. Information such as page hits, frequency and related statistics contrasted against each other can also show which applications, or portions thereof, have consistently good (or bad) performance. Management reports generated from the collected raw data can provide insights on the volume of users that pass though the application.

There are fundamentally two ways to approach problem solving in a production environment:

1. One is through continual data collection through the use of application monitoring tools that, typically, provide up-to-date performance, health and status information.

2. The other is through trial and error theorizing, often subject to whatever data is available from script files and random log parsing.

Not surprisingly, the latter approach is less efficient, but it's important to understand its other drawbacks as well. Introducing several levels of logging to provide various types of information has long been a popular approach to in-house application monitoring, and for good reason. Logging was a very trusted methodology of the client-server era for capturing events happening on remote workstations to help determine application problems. Today, with browsers dominating the thin client realm, there is little need for collecting data on the end user's workstation. Therefore, user data is now collected at centralized server locations instead. However, with the general assumption that all possible points of logging are anticipated and appropriately coded, data collection on the server is also problematic. More often than not, logging is applied inconsistently within an application, often added only as problems are encountered and more information is needed.

In contrast, application monitoring tools offer the ability to quickly add new data -without application code changes - to information that is already being collected, as the need for different data changes with the ongoing analysis.

While logging worked well in the single user environment, there are some inherent problems with logging in the enterprise application server environment:

• Clustered environments are not conducive to centralized logs. This is a systemic problem for large environments with multiple servers and multiple instances of an application. On top of the problem of exactly how one is to administer the multiple logs, is the user's ability to bounce around application servers for applications that do not use HTTP Session objects. Coordinating and consolidating events for the same user spread across multiple logs is extremely difficult and time consuming.

• Multiple instances of applications and their threads writing to the same set of logs imposes a heavy penalty on applications that essentially spend time synchronized in some logging framework. High volume Web sites are an environment where synchronization of any kind must be avoided in order to reduce any potential bottlenecks that could result in poor response times and, subsequently, a negative end user experience.

• Varying levels of logging requires additional attention: when a problem occurs, the next level of logging must be turned on. This means valuable data from the first occurrence of the problem is lost. With problems that are not readily reproducible, it's difficult to predict when logging should be on or off.

• Logs on different machines can have significant timestamp differences,
making correlation of data between multiple logs nearly impossible.

• Beyond the impact of actually adding lines of code to an application for monitoring, additional development impacts include:

o Code maintenance: The functionality, logical placement and data collected will need to be kept up, hopefully by developers who understand the impact of the code change that was introduced.

o Inconsistent logging: Different developers may have drastically different interpretations of what data to collect and when to collect it. Such inconsistencies are not easily corrected.

o Developer involvement: Involving developers in problem determination becomes a necessity with log-based approaches, since the developer is usually the best equipped to interpret the data.

• Application monitoring accomplished through coding is rarely reused. Certainly the framework itself can be reused, but probably not the lines of code inserted to capture specific data.

• When logging to a file, the impact on the server's file I/O subsystem is significant. Few things will slow down an enterprise application more than writing to a file. While server caches and other mechanisms can be configured to minimize such a hit, this is still a serious and unavoidable bottleneck, especially in high volume situations where the application is continually sending data to the log.

• While Aspect-Oriented Programming is proving a valuable technology for logging, it has yet to be embraced by the technical community.
Not surprisingly, it is also common for development teams to try to collect basic performance data using their logging framework, capturing data such as servlet response time, or the timings of specific problematic methods, etc., in order to better understand how the application performs. This activity is victim to the same disadvantages mentioned above, in that any suspected problem points are correctly identified and instrumented. If new data points are identified, then the application must be modified to accommodate the additional data collection, retested and then redeployed to the production environment. Naturally, such code also requires continual maintenance for the life of the application.

The benefits of a proactive, tool-based approach to application monitoring are many:

• No code

This, by far, is the single most valuable benefit regarding a tools-based approach. Application monitoring tools allow for the seamless and invisible collection of data without writing a single line of code.

• Fewer developer distractions

With application monitoring no longer a focal point, developers can instead concentrate on the logic of the application.

• Reusability

Application monitoring tools are written to generically capture data from any application, resulting in a tremendous amount of reuse built into the tooling itself. Without doing anything extraordinary, an application monitoring tool can capture data for a variety of applications as they come online.

• Reliability

While you should still perform due diligence to ensure that a tool is working properly in your environment, application monitoring tools from major vendors are generally subject to extensive testing and quality assurance for high volume environments.

• Understandable results

Consolidation of data occurs at some central console and the results can be readily understood by a systems administrator. Only when the system administrator has exhausted all resources would developers need to assist in troubleshooting by examining data from a variety of subsystems.

• Cost

While there is the initial expenditure of procuring such a tool, there is also the very real possibility of eventual cost savings - particularly in terms of time.

In general, application monitoring can be divided into the following categories:

1. Fault

This type of monitoring is primarily to detect major errors related to one or more components. Faults can consist of errors such as the loss of network connectivity, a database server going off line, or the application suffers a Java out-of-memory situation. Faults are important events to detect in the lifetime of an application because they negatively affect the user experience.

2. Performance

Performance monitoring is specifically concerned with detecting less than desirable application performance, such as degraded servlet, database or other back end resource response times. Generally, performance issues arise in an application as the user load increases. Performance problems are important events to detect in the lifetime of an application since they, like Fault events, negatively affect the user experience.

3. Configuration

Configuration monitoring is a safeguard designed to ensure that configuration variables affecting the application and the back end resources remain at some predetermined configuration settings. Configurations that are incorrect can negatively affect the application performance. Large environments with several machines, or environments where administration is manually performed, are candidates for mistakes and inconsistent configurations. Understanding the configuration of the applications and resources is critical for maintaining stability.

4. Security

Security monitoring detects intrusion attempts by unauthorized system users.
Each of these categories can be integrated into daily or weekly management reports for the application. If multiple application monitoring tools are used, the individual subsystems should be capable of either providing or exporting the collected data in different file formats that can then be fed into a reporting tool. Some of the more powerful application monitoring tools can not only monitor a variety of individual subsystems, but can also provide some reporting or graphing capabilities.

One of the major side benefits of application monitoring is in being able to establish the historical trends of an application. Applications experience generational cycles, where each new version of an application may provide more functionality and/or fixes to previous versions. Proactive application monitoring provides a way to gauge whether changes to the application have affected performance and, more importantly, how. If a fix to a previous issue is showing slower response times, one has to question whether the fix provided was properly implemented. Likewise, if new features prove to be especially slower than others, one can focus the development team on understanding the differences.

Historical data is achieved by defining a baseline based upon some predefined performance test and then re-executing the performance test when new application versions are made available. This baseline has to be performed on the application at some point in time and can be superceded by a new baseline once performance goals are met. Changes to the application are then directly measured against the baseline as a measurable quantity. Performance statistics also assist in resolving misconceptions about how an application is (or has been) performing, helping to offset subjective observations not based on fact. When performance data is not collected, subjective observations often lead to erroneous conclusions about application performance.

In the vein of extreme programming, one is urged to collect the bare minimum metrics and thresholds which you feel are needed for your application, selecting only those that will provide the data points necessary to assist in the problem determination process. Start with methods that access backend systems and servlet/JSP response timings. Prepare to change the set of collected metrics or thresholds as your environment evolves and grows.

There are two main factors measured by end user performance tools: availability and response time.

The first is measured by the uptime of the enterprise applications. Response time measurement looks at the time to complete a specific job - starting at the end users' desktops, through the network, the servers, and back.

End user performance management typically starts from a lightweight agent installed on the end user's computer. The agent records network availability, response time, delay, and occasional failures of requests initiated by the end user in real time. This data is forwarded to a central database. It then performs trend analysis by comparing the real-time data collected by the agents with historical patterns stored in the database. Reports are then generated to display the number of important measures such as transaction time, delays and traffic volume.

Response time has always been a key component in the measurement of performance. In this era of networks and rapid deployment of applications, the quest for end-to-end response time has become legend. Unfortunately, most of today's application performance solutions are more myth than substance.

There are two fundamental approaches to the problem:
- Using various proxies, such as ping
- Observing and measuring application flows.

Most response time metrics turn out to be derived values using simple ping tools. Ping is an invaluable tool, but it has severe limitations as a response time measure. Pinging routers is problematical because of questionable processing priority and consequently, reliability of the measurement. If you are pinging servers, how does response time vary with processing load and other factors? Vendors may claim they have other measures, but most are ping variants, perhaps with a slight improvement in reliability over classic ping. If ping is used, then the value derived must be used properly, as part of a series of measurements over time.
As an alternative response time measurement, some monitoring/probe products apply cadence or pattern-matching heuristics to observed packet streams.

These provide a measurement of apparent response time at the application level but this means deploying multiple, relatively expensive probes to analyze the packet stream. Existing RMON and SNMP standards do not cover this area, so all solutions rely on proprietary software to collect and report on the data. Other concerns are the quality of the heuristics, the scalability of the solution and the continuity of support across a product's lifetime.

As more and more enterprise applications are running in distributed computer networks, the loss of revenue due to down time or poor performance of the applications is increasing exponentially. This has created the need for diligent management of distributed applications. Management of distributed applications involves accurate monitoring of end-user service level agreements, and mapping them to the application level, system level, and network level parameters. In this paper, we provides a statistical analysis on mapping application level response time to network related parameters such as link bandwidth and router throughput by using some simple queueing models.

With more and more enterprises running their mission-critical e-business applications in distributed computing net-works, the effective management of these applications is crucial for the success of business. The loss of revenue due to downtime or poor performance of these distributed applications increases exponentially. Distributed applications operate in a very different environment compared to client/server applications.

In client/server paradigm, the components of a software application are shared between client and server computers. In a distributed computing environment, the application can have their components running on many computers across an entire network. The distinction between the client and server disappears. Normally, a component in a distributed application acts as both client and server. A distributed application is an intelligent control entity that can be of any type that runs in a distributed environment. It can be a single component such as a web page, a database, a reusable component, a URL, a UNIX process, a Java class or EJB, etc. But theoretically, a distributed application is a combination of objects and processes with some dependent relationships that communicate with each other in order to provide a service to end users.

Monitoring solely from the client's side is another class of techniques. In contrast to the methods mentioned so far it is possible to measure the actual time an application needs to complete a transaction, i.e. it is metered from a user's perspective. Nevertheless, this class of techniques still suffers from one general problem: it is possible to detect an application's malfunction in the moment it happens but it still does not help in finding the root cause of the problem. Therefore in general this class of techniques is only useful to verify fulfillment of SLAs from a customer's point of view, but additional techniques have to be used for further analysis in order to detect the reason of a QoS problem. There are two basic methods for monitoring performance from a client's perspective: synthetic transactions and GUI based solutions.

The synthetic transactions method uses simulated transactions in order to measure the response time of an application server and to verify the received responses by comparing them to previously recorded reference transactions. Several simulator agents, acting as clients in a network, send requests to the application server of interest and measure the time needed to complete a transaction. In case response time exceeds a configurable threshold or the received server response is incorrect in some way, the agents usually inform the manager by generating events. As solely synthetic transactions are monitored and not real transactions initiated by actual users, this technique is only useful to take a snapshot of a server's availability, but not to verify the fulfillment of service level agreements. To get measurement data close to actual user experience, the interval between simulated transactions has to be reduced to a minimum. As a consequence the application service could experience serious performance degradation. Further problems arise from agent deployment in large networks.

The GUI based approach meters the actual user transactions but to avoid the need for accessing the client application's source code, a new approach was recently developed: As every user request both starts and ends with using/changing a GUI element at the client side (e.g. clicking a web link and displaying the appropriate web page afterwards), simply observing GUI events delivers the needed information about start and end points of user transactions. A software agent installed at client site devices gathers the transaction data of interest from a user's point of view. The advantages of this technique are that the actually occurring transaction duration is measured and that it can be applied to every application service client. Furthermore, only very little performance impact is caused on the monitored application. However, there seems to be two major problems. First of all, mapping GUI events to user transactions is a difficult task regarding non-standard applications and therefore requires additional effort by the administrator. Secondly, there are few agents that use this technique.

As mentioned before, client–based monitoring cannot identify the reason for performance degradation or malfunction of an application. Therefore solutions that monitor both from the client– and from the server–side are necessary. As details about the application and problems within the application cannot be gathered externally, these approaches rely on information supplied by the application itself. Our studies have shown two basic classes that allow application–wide monitoring. These are application instrumentation and application description.

Application instrumentation means insertion of specialized management code directly into the application’s code. The required information is sent to management systems by using some kind of well–defined interface. This approach can deliver all the service–oriented information needed by an administrator. The actual status of the application and the actual duration of transactions is measured and any level of detail can be achieved. Subtransactions within the user transactions can be identified and measured. However, application instrumentation is not very commonly used today. This is mainly due to the complexity and thus the additional effort posed on the application developer. The developer has to insert management code manually when building the application. Subtransactions have to be correlated manually to higher–level transactions. As the source code is needed for performing instrumentation, it definitely has to take place during development

Examples for approaches using application instrumentation are the Application Response Measurement API (ARM) jointly developed by HP and Tivoli and the Application Instrumentation and Control API developed by Computer Associates. Both approaches have recently been standardized by the Open Group. ARM defines a library that is to be called whenever a transaction starts or stops. Subtransactions can be correlated using so called correlators. Thus the duration of the transaction and all subordinate transactions can be measured. AIC in contrast was not explicitly developed for performance measurement but might be used in this area as well. It defines an application library to provide management objects that can transparently be queried using a client library. Additionally, a generic management function can be called through the library and thresholds of certain managed objects can be monitored regularly. Both ARM and AIC suffer from all the problems mentioned above and thus are not in widespread use today.

As most of the applications in use today somehow deliver status information but are not explicitly instrumented for management, application description techniques can be used. As opposed to the instrumentation approach, no well–defined interface for the provisioning of management information exists. The description therefore comprises where to find the relevant information and how to interpret it. Examples might be scanning of log files or capturing status events generated by the application. The major advantage of application description techniques is that it can be applied to legacy applications without requiring access to the source code. It can be done by a third party after application development, while the reasonable approach again is to provide the description by the developer. Application description faces two major problems: The information available typically is not easy to map to the information needed by the administrator. Especially in the area of performance management typically only little information is available. Moreover monitors are needed to extract the information from the application. Only very little information can be gathered by standard monitors and thus specialized monitors must be developed for every application. The most prominent representative of application description suited for performance monitoring is the Application Management Specification (AMS). Most other approaches, like the CIM Application Schema mainly focus on configuration management. An example for a tool making use of application description is Tivoli Business System Manager, which reads in AMS based Application Description Files (ADF) to learn about the application or business system to be managed.

Conclusion

The root cause of application performance is not a trivial question. Software code, system architecture, server hardware and network configuration can all impact an applications performance. These tools are developed to provide specific information on how the system as a whole is behaving.

Application Performance tools will determine the client and server's performance,
network bandwidth and latency. They provide a map of the conversation where suspect code can be evaluated. The most sophisticated tools in this area have a high level of drill down capabilities and can provide information gathered from both Client and Server or Server-to-Server conversations. Network Administrators as well as Development may use these tools. Slow user response time can be costly and even though it may be a small part of the user day, it can really add up when it impacts hundreds of users.

Monitoring a variety of application metrics in production can help you understand the status of the components within an application server environment, from both a current and historical perspective. As more back end resources and applications are added to the mix, you need only to instruct the application monitoring tool to collect additional metrics. With judicious planning and the right set of data, proactive monitoring can help you quickly correct negative application performance, if not help you avoid it altogether.

Proactive monitoring provides the ability to detect problems as they happen, and fix them before anyone notices. If problems are going to happen, its better that to find them before the customer does.

another dated summary of a SOA conference

On Thursday, March 2, 2006, I attended the IBM SOA Architecture Summit at the Renaissance Hotel in Washington DC. The following paragraphs summarize some of the ideas that were presented as well as some of my thoughts. I have made no effort to consolidate these thoughts as time has not permitted. I do not explain acronyms as I expect people who might read this to already know the acronyms. This paper (a) serves to help me remember what I thought of yesterday and what was presented, and (b) serves to inform others at a high level what was covered in the summit.

Overall, the summit was good. I felt like it was a little long given that much of the day we didn’t get into what I would call the nuts and bolts of IBM’s offerings although the presenters indicated that they had. The speaker I enjoyed the most was a man who used to be a professor in computer science. I liked him the most because my academic background is in software engineering and that was the perspective from which he spoke. I got the most out of the early sessions as I was not quite as alert right after lunch. The summit closed with a Q&A session where I was able to get in the first question. My question was where IBM was in regards to the WS-BPEL Extension for People specification. I indicated that it was my perception that at the current time this specification mostly consisted of a white paper concept and I wondered when the specification might be finalized. They said that the standards committee on which IBM sits should complete the specification in 4-6 months and that it would be another 4-6 months before it was productized. Other questions that came up from attendees included:

How might contractors be given incentive to adopt SOA?
How could Government executives be sold on SOA?
How would one go about selecting a pilot SOA project?

As explained above, the following are random thoughts, observations, and points that I either had or that were made during the summit. Hopefully, if you are reading this, you will gain some kind of insight.

70-80% of most organization’s IT budgets is spent on maintenance.

In The World is Flat, a book that seems to be a favorite among Government workers, Thomas Friedman says”

"We are taking apart each task and sending it around to whomever can do it best, and because we are doing it in a virtual environment, people need not be physically adjacent to each other, and then we are reassembling all the pieces back together at headquarters or some other remote site. This is not a trivial revolution. This is a major one. These workflow software platforms enable you to create virtual global offices - not limited by either the boundaries of your office or your country - and to access talent sitting in different parts of the world and have them complete tasks that you need completed in real time."

In a report dated March 5, 2005, Gartner reflected, “Point to point interfaces result in an ever increasing burden.” (Duh!)

IBM’s SOA lifecycle can be summarized: Model, Assemble, Deploy, Manage, and Govern

A difference between BPM and BPR is the formers ability to model and analyze metrics before investing in actual development. In other words, the reason BPR was deemed such a failure is that months (and years) were spent engineering systems to replace older ones and by the time these systems were deployed, it was determined that the new processes were less than adequate. There was no support for capabilities such as process simulation as there is with today’s BPM products. Today’s BPM products allow rapid change and are agile and flexible.

SOA is much more than software. It includes governance, best practices, and education – among others.

IBM stated that we are past the disillusionment cycle on Gartner’s Hype chart.

Incidentally, one of the presenters stated that all of IBM’s software development was performed by AMS.

Distinct components that comprise architectures include: Systems, Data, Applications, Processes, Networks, Storage, and Standards. Each of these components should be loosely coupled.

In implementing pilot SOA projects, look for manageable project entry points. Avoid the “big bang” approach. Then: figure out the target reference architecture, develop a roadmap, and apply governance.

SOA has at least four important perspectives:

Business – SOA can be seen as Capabilities packaged as a set of services.
Architecture – SOA can be seen as an engineering Style, consisting of a Service Provider, Service Requestor, and Service Description.
Implementation – SOA can be seen as a Programming Model with standards, tools, methods, and technologies.
Operations – SOA can be seen as a Set of Agreements among Service Providers specifying Quality of Service, and key business and IT metrics.

Three principal SOA goals and benefits: (1) Separate specification from implementation, (2) Use higher abstraction from child to parent (as opposed to OOA/OOD which introduced the concept of inheritance from parent to child – which introduced other problems), Follow loose coupling. These three qualities allow for reusability, flexibility, and agility.

Five ways that SOA has transformed IT: (a) more standards-based, (b) greater degree of interconnectedness (while simultaneously providing for decoupling – in other words, greater interoperability) between components, (c) greater reusability, (d) greater focus on business rather than IT, (e) greater organizational commitment.

One of the differences between services and processes is that services are atomic and composite.

IT architecture can be thought of as consisting of five layers:
Consumers
Processes
Services
Service Components
Operational Systems

In this five layer view, Consumers and Processes are Implementation-related (and Consumer-based) while Service Components and Operational Systems are Specification-related (and Provider-based). Thus, the Services Layer separates the Specification Layers from the Implementation Layers.

A major element that was missing from the basic OOA/OOD/OOP world was that of having objects “talk” to one another. This created too many interfaces as we can see from the many spaghetti-drawn software architecture diagrams. The introduction of the ESB serves this purpose.

Three elements of Business-Driven Development: SOA (provides flexibility and reuse), Model Driven Architecture (provides efficiency and quality), Business Innovation and Optimization (provides for responsiveness and (I will add) Competitive Advantage)

For examples of patterns, go to http://www-128.ibm.com/developerworks/rational/products/patternsolutions

Another layered approach can be seen below (remember the old OSI seven layer view?):
Layer One Network layer (i.e. http, smtp)
Layer Two XML (i.e. Infoset, namespace, schema)
Layer Three Service Description (i.e. WSDL, RAS)
Layer Four Invocation and Messaging (i.e. WSI, SOAP)
Layer Five Service Discovery (i.e. WSIL, UDDI, RAS)
Layer Six Service Orchestration (i.e. BPEL)

Today there are two kinds of developers: application and integration. Integration Development consists of both and primarily Information/Data Integration and Message Integration.

Orchestration and Choreography separates business logic (business rules) from control logic.

IBM’s tools allow you to move from Requirements Management (Requisite Pro) to a process model and, in turn, process models can be turned into UML. A tool included in this process includes Software Architect in addition to Rational Rose. From their development platform, one can alternate between different views: Software Architect, Application Developer, Integration Developer, Business Modeler all in a single UI which is based on Eclipse, an open source development UI.

Service elements to consider when thinking about the concept of coupling include: services, messages, interfaces, contracts, policies, conservations, states, transactions, and processes.

Why is SOA a big deal? Because it provides for a self-describing component that provides services based on standards. We have had these before but not all at the same time.

Business Processes can be managed through BAM, Dashboards, KPIs (Key Performance Indicators), etc.

Eleven elements of Governance: Incentives, Mission, Domains, Enterprise Architecture, Technical Standards, Development and Deployment, Operations, Roles, Organizations, Processes, and Shared Services.

Shared Services leads to the establishment of authority:
Authoritative Services
Authoritative Data Sources
Authoritative Semantic Meaning of Data and Services

In establishing a process framework, think of:
What has to be done?
What is scope of policies?
How will it be accomplished?
Who has authority?
When is oversight and control provided?
Where is governance enforced?
How is capability measured?

IT & SOA governance needs to come together. The most important best practice is for the CIO to report to executive leadership.

In laying out an SOA roadmap (this is an iterative process):

First, what is the scope and what are the pain points?
Second, assess current capability
Identify gaps between desired capability and current capability
Create a roadmap
Execute


Four possible entry points for an SOA: create services from new or existing application (this is NOT an SOA – just code), implement a SOA pilot (preferred initial entry point), LOB process integration, and enterprise business and IT transformation (if you start here you will probably fail). Suggestion: start at possibility two and proceed through to possibility four.

Degrees of integration include (in order – BTW I missed one but I thought this was an interesting continuation): silo, integrated, componentized, virtualized, dynamically reconfigured

IBM used GCSS-AF as an example of a successful SOA project!

IBM recommended that all organizations establish an SOA Center of Excellence.

The TAFIM has evolved into the TOGAF. I am familiar with FEAF, DODAF, etc. but not TOGAF.

IBM states that functionality should be subordinate to architecture. One of the audience members asked during the Q&A function how this can be mandated since people’s careers are based on functionality?

one of the images I discussed in previous post (SDLC)