trip report describing two recent Gartner conferences

From June 10 to June 15, I attended two consecutive Gartner conferences in Nashville. In the paragraphs that follow I attempt to share some of my observances in the areas of (a) application architecture, development and integration, and (b) enterprise architecture.

The contents below can be grouped into the following topics:
• General overview
• Web Oriented Architecture
• Web 2.0
• Web 3.0 and Web 4.0
• Random Observations
• Some SOA Best Practices
• Websites worthy of review
• Various statistics relevant to IT

General overview

The conference was interesting, not least of which Gartner predicts that SOA is already becoming legacy and among many other topics there is a new buzzword (since 2006, I believe) which is WOA (Web Oriented Architecture). One thing I did get some clarity on is that there is no such thing as a SOA architect! This may be obvious to you, the reader, but in the past year I have repeatedly heard this label used almost as a job position. SOA is simply a design pattern! The correct terminology is to speak of a solutions architect who employs among other design patterns, the SOA design pattern. ESB is also simply a design pattern. There was some discussion that ESBs may impede progress within the SOA strategy and it was forecasted that the ESB may even start to become obsolete within the next two years. Speaking of obsolescence, some analysts are actually even starting to consider AJAX as already becoming obsolete (although it will still be THE major technology for rich internet applications for the next couple of years). I was alerted to the importance in learning TOGAF in addition to FEA and DODAF.

Web Oriented Architecture

Web-oriented architecture (WOA) is based on the principles of REST. Representational state transfer (REST) is the style of the Web, and its principles should be followed for any SOA project.
What is the architecture of the Web, and how is it different from SOA? Although there are many formal definitions of SOA, the market is coalescing around a practical meaning for the term, which includes the following: SOA is about applications; while service orientation is a concept that can be applied to many things (storage area network is a service-oriented approach to storage), which does not make it SOA. SOA is about how one designs applications as the providers and consumers of a set of defined, agreed-upon interfaces (services). In this way, services are a unit of modularity in the same way that subroutines, objects and components are units of modularity. Modularity is how we have improved the ability of programs and our ability to create more complex and integrated software. A key concept in the current modularity that SOA provides is that it incorporates encapsulation. In other words, the service interface is the only means of accessing anything about that service or the information it manages. A key attribute of these new modules is that they are shared, not reused. This means that the definitions that constrain their operation are shared as well. Another thing that is critical to the value of SOA is the fact that the only thing these modules must have in common is the network. That network — a Simple Object Access Protocol (SOAP) network or its logical equivalent — is the only means of access. This removes any dependency on development time components. The definition of a service is not its WSDL or its implementation code, but how it is manifested over the wire, which eliminates any technological dependency on anything except the application wire protocol. The final assumption is that modules are mostly the same as a subroutine or other structured code component. Thus, some say that current SOA initiatives follow a different architectural pattern than the Web and, therefore, cannot be expected to deliver some of the benefits that are inherent in the Web-oriented pattern.
The only real difference between traditional SOA and the concept of WOA is that WOA advocates REST, an increasingly popular, powerful, and simple method of leveraging HTTP as a Web service in its own right (and carefully devised by the co-creator of HTTP, Roy Fielding.)
WOA itself is a reflection of the classic Reach vs. Rich argument, in that richness is an outcome of robustness and complexity but cuts down how much reach you have to others, particularly in heterogeneous communities. Some say that the modern SOA technologies and architectures have become a near-morass of complex standards, protocols, and products, and that the result is that the present list of WS-* standards is nearly incomprehensible.
This is making some people fall back to simpler and more straightforward methods that work, while hiding the protocol complexity in the application state. Some say that one doesn’t usually need anything more complex than XML and HTTP, one of the most scalable, proven, and widespread protocols on the planet, along with HTTP’s verbs: GET, POST, PUT, DELETE.
REST does not handle enterprise issues like two-phase commit, messaging, and asynchronicity. WOA is definitely not the hammer for every job. Certain applications, particularly in the high-end of the enterprise, require the more sophisticated portions of the SOA stack (don’t forget, WOA is a subset of SOA and is not disjoint as long as you follow the principles of service orientation). However, for many uses, WOA is the most interoperable, easiest to implement, and most highly scalable technique for building open web services.

Web 2.0

Much of the conference was centered on Web 2.0. One of the languages used in AJAX web application programming is "JavaScript Object Notation" (JSON) which is pronounced like the English name Jason. JSON is based on a subset of the JavaScript Programming Language. JSON is a fast, compact representational format that has advantages when used with JavaScript (as is found in browsers). XML, when used with browsers, must be parsed and processed by JavaScript while JSON is simply "consumed" directly. This lightweight data-interchange format is easy for humans to read and write, and easy for machines to parse and generate. JSON is a text format that is completely language independent but uses conventions familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON an ideal data-interchange language. It provides a simple alternative to XML for asynchronously transmitting structured information between a client and its server. In December 2005, Yahoo! began offering some of its Web Services optionally in JSON. Google started offering JSON feeds for its GData web protocol in December 2006.

Web 3.0 and Web 4.0

Have you ever wondered what the Internet will look like in the future? Most people have heard of some aspect of what has been called web 2.0. You know things like social networking, lightweight programming, web as a platform, and collective intelligence. Most people recognize these by their street names Facebook, AJAX, Google Maps, and wikipedia respectively.
Many people refer to Web 3.0 as the semantic web. The semantic web is where machines read web pages as humans do today. A web where search engines and software agents peruse "the Net" and find us what we're looking for. Some of the technologies required to perform such tasks exist today including the Resource description Framework (RDF) and the Web Ontology Language (OWL). These work by adding all sorts of machine readable metadata to human readable web pages.
The web 2.0 technologies are beginning to trickle into real-world sites, service experiences, and tools. Examples of these technologies can be seen with Yahoo!'s food site, which is driven by semantic metadata, the semantic portal developed by Radar Networks, the Jena development platform being creating at HP, and the Oracle Spatial database.
Just as much of life, the semantic web is a good-news, bad-news thing. You can do all these wonderfully complex queries, but it takes a tremendous amount of time and metadata to make it happen. This is due to having to completely re-annotate the entire Web in order to make existing web pages machine readable.
For this reason, many researchers opt for a very different approach to the semantic web. Instead of an overhaul of Web formats, they're building agents that can better understand the Web pages of today. The pages aren't easier to read, but rather the software agents are getting smarter.
There is yet another view of the semantic web taken by so called "semantic searchers". Rather than providing automatic information retrieval, semantic search engines seek to improve on the Google-like search model that we've grown so accustomed to. the idea is to move beyond mere keyword searches to a better understanding of natural-language queries. This type of natural-language processing has been in development for years and now is beginning to find its way onto the public Web.
One could argue, however, that while many associate Web 3.0 and the Semantic Web, the two are far from synonymous. There are countless other concepts ready to alter our online experiences of the future. Many of these concepts go beyond semantics using space, images, and sound.
One such concept is the so called 3D Web, a Web you can walk through. This has been seen as an extension to the "virtual worlds" popping up on today's Web. It has been said that the Web of the future will be the big alternate universe reminiscent of Second Like and There.com.
As with everything in like there are those who scoff at this notion of the 3d Web. They see the 3D Web as a re-creation of our existing world. On the 3D Web, you can take a virtual tour of an unfamiliar neighborhood and shop for houses or visit famous sites you've never seen. Google earth already offers a similar experience. The problem is that 3D only goes so far and doesn't enhance the 2D world of text, pictures, and video. An interesting idea beginning to emerge is that of a media-centric Web which offers not only language-based search but pure media search. Today's searching mechanisms rely primarily on keywords, even when searching for images, videos, and songs, this is a woefully inadequate approach. Some companies are looking to reinvent media search, hinting at a world where we search for media with other media, not just keywords.
Speaking of web 2.0 and web 3.0, some are already talking about a web 4.0, dubbed the ubiquitous (or pervasive) web. The Web already reaches beyond the desktop, to cell phones and handhelds, but who says it has to stop there? What about having the windows in your home open automatically when the temperature changes? By leveraging mesh networks, wireless networks consisting of tiny nodes that can route data to and from almost anywhere, the possibilities are endless. However, regarding web 2.0, web 3.0, and web 4.0, perhaps we should all avoid getting into a naming competition.

Random Observations

Ruby is an open-source all-purpose, object-oriented scripting language that was developed in Japan in 1995. Rails is a open-source, full-stack, Model-View-Controller (MVC) web framework, designed using the Ruby programming language. Because of Ruby's dynamic nature, Rails provides many exciting features for programmers, such as easy database access and methods to utilize some of the newest technologies on the web.
Service Component Architecture (SCA) is another significant new development. Many of the vendors in the consortium that created it (www.osoa.org) have products in the pipeline that conform to SCA. However, because of the breadth of capability, and the sheer number of vendors involved, Gartner expects the formal ratification process to take a long time. They also expect that this will suffer from the usual challenges with vendor-led standards, in that many vendors will create implementations that conform to the standard, but that also contain significant extensions that are useful to customers. This will leave organizations with the choice between retaining interoperability, and conforming to the standard at the cost of additional programming time or by using these convenience features and accepting limitations on the interoperability.
Start to look for higher-level standards, such as WS-SX and WS-RX, within the near term, as they become mature (WS-SX = Web-Services Secure Exchange; WS-RX = Web-Services Reliable Exchange).
“Net Gen” kids now entering workforce and consumer market and will demand web 2.0 capabilities, potentially having a large impact on corporations. There are 80 million people ages 8 to 28 in U.S. alone.
SOAs don’t finish; it’s a journey, not a destination.
Three main ways to approach SOA: REST/POX, web services, and proprietary methods.
English has been dominant language in blogs but Japanese is about to catch up or pass English as dominant language.
Blogs can be particularly helpful for technology evangelism, project management, standards work, technology change management. Social networking is useful in finding needed expertise (i.e. LinkedIn, myspace, facebook). For instance, one could use facebook or myspace in a corporation to describe specific projects, technologies, skills, etc. One idea is to make a matrix of stakeholders vertically and horizontally and brainstorm on potential reasons why such people might collaborate.
Mashups are predicted to become dominant form of composite applications. These do not have to be limited to map data. A mashup is simply the idea of bringing together two or more different data sets.
AJAX is not new. Pioneers in AJAX technology include outlook web access (circa 1998) and desktop.com (circa 1999).
BPM really has very little to do with IT; its mostly a management discipline.
TOGAF is the defacto enterprise architecture framework standard. TOGAF provides specific methods in its ADM. TOGAF is frequently applied to SOA.
WS-I provides ws-i conformance tools. Can be downloaded from WS-I site. Mindreef, IBM, Microsoft, Parasoft all have versions of these tests.
Aspect-oriented programming (AOP), and aspect-oriented software development (AOSD) are gaining momentum. See wikipedia for more information.
In late 2006, Target was sued by National Federation for the Blind for having a web site inaccessible to the blind. Results could have large impact.
Microsoft introduced Silverlight about 2 months ago to compete against Adobe’s Flash.
About two weeks ago Microsoft released a tool to allow non-IT people to build mash-ups.
Mission critical applications deployed as a mashup is risky as its dependent on external data.
There are 100+ AJAX vendors, including tools and vendors Adobe Flash, Eclipse RCP, Microsoft WPF and Silverlight, and IBM Expeditor.
Sun has introduced JavaFX to compete with Flash and Silverlight.
There appears to be no correlation between EA maturity and how long one has been doing it. Architecture impact remains low. This indicates that there a lot of companies are doing lots of work but it is not making any difference.
Try to avoid modelitis (the compulsive drawing of lots of pretty models) even though it is fun to play with cool tools.
Avoid “management by magazine”.
Know standard ISO/IEC 42010 or ANSI/IEEE 1471-2000
DoDAF, FEAF, TOGAF, Gartner, Zachman are all examples of architecture frameworks.

Some SOA Best Practices

Start by looking at REST/POX. If a problem can be solved via REST/POX, then focus there. If the problem needs more capabilities than REST/POX provides, then move up to Web services, and eventually to proprietary approaches, as necessary. This may seem to increase the complexity of a single solution, but in reality, it simplifies the overall solution. Many features will be implemented with the simplistic REST/POX approach, which will reduce the number of complex proprietary approaches and more-complicated Web services approaches. Use the right tool for the right job, and ensure that your toolkit includes REST/POX, Web services and proprietary approaches.
Today's programmatic access uses a combination of REST/POX, Web services and proprietary approaches. For new development, start simple and step up as required.
Create a service when:
• Crossing a boundary
• Seeking more than one reuse scenario
• Providing a significant business process
• Invoking a more-complex process
• Requesting information
• Exposing a high-level API
• Exposing public data
• Needs to be shared
• Needs to be explicitly defined and described
• Changing frequently
• Describing a discrete function
• You want capabilities instead of features
Avoid a service when:
• Performance is paramount
• Working within one system
• Transferring large data sets
• Executing transactions
• Data is of extremely fine granularity

Websites worthy of review

http://www.writely.com
http://www.riya.com
http://iscrybe.com/main/index.php
https://www.youos.com
http://www.zoho.com
http://www.pageflakes.com
http://googlesuggest.com
http://tiddlywiki.com
http://secondlife.com
http://www.socialtext.com
http://www.goowy.com
http://www.digg.com

Various statistics relevant to IT

• Gartner predicts that SOA will be partly used in more than 80% of new, mission-critical applications and business processes designed by 2010.
• Through 2009, organizations that do not consider data integration and data quality as critical components of SOA will double their risk of failure in SOA-related projects (0.8 probability).
• Through 2008, the majority of Global 1000 companies will adopt the technology-related aspects of the Web 2.0 phenomenon, but fail to adopt the aspects of Web 2.0 that have a social dimension. The result will have a minimal business impact (0.6 probability).
• Organizations that align their BPM and SOA initiatives in 2007 will double their likelihood of becoming industry leaders by 2011 (0.7 probability).
• Web architectural style will be used in more than 75% of new applications by 2012 (0.8 probability).
• By 2012, 75% of practitioners of the current style of SOA will have failed to measurably improve their application agility (0.8 probability).
• By 2012, current forms of service-oriented architecture (SOA) will be considered legacy architectural styles (0.9 probability).
• By 2010, Web mashups will be the dominant model (80%) for the creation of composite enterprise applications (0.7 probability).
• The high cost and complexity of information integration continues to eat at 30% to 40% of most corporate IT budgets.
• Organizations building an enterprise architecture will achieve marginal to no benefits during the first three years following its implementation without explicit links to software process and behavioral change (0.8 probability).
• By 2010, 40% of existing EA programs will be stopped (0.9 probability). Failure to define value leads to a failure to deliver value, which leads to a failure of confidence, which leads to EA program failures.
• Fully mature semantic reconciliation tools will not be available until 2011 (0.7 probability). By the end of 2009, 40% of a global company's data will be defined in some way by XML (0.7 probability).
• By the end of 2009, 75% of the Fortune 500's interapplication messaging infrastructure will be formatted in XML (0.7 probability).
• By 2010, web-oriented architecture will account for 60% of SOA development in the enterprise (0.7 probability).
• Through 2010, lack of working SOA governance arrangements will be the most common reason for SOA failure (0.8 probability).
• SOA is currently at trough of disillusionment. 2-5 years mainstream adoption.
• By 2010, about $1 of every $3 spent on IT professional services will be spent on a solution using SOA, Web services or Web 2.0 (0.8 probability).
• According to Gartner Dataquest, in 2006, the worldwide market for IT professional services involving Web services or Web 2.0 was $58.9 billion, and the market will grow at a compound annual rate of 27% to reach $152.4 billion by 2010. Growth will be fastest in development and integration services through 2009 pending the availability of a critical mass of commercially available SOBAs.
• Through 2009, lack of SOA, Web services and Web 2.0 skills will require enterprises to obtain expertise from service or software providers for 70% of their projects involving these technologies (0.8 probability).
• Through 2010, fewer than 25% of large companies will have developed the technical and organizational skills needed to deliver enterprisewide SOA (0.8 probability).
• By 2008, 50% of companies will use Web 2.0-style services in their SOA initiatives (0.8 probability).
• Gartner predicts that SOA will be partly used in more than 80% of new, mission-critical applications and business processes designed by 2010.
• By 2011, at least 80% of all commercial software solutions will include elements of open source (0.8 probability).
• By 2010, WOA will account for 60% of SOA deployment in the enterprise (0.7 probability).

This is a co-worker's brother-in-law (see article below). He is all over the place, including at last week's Gartner's SOA conference that I attended. I will be writing a trip report as time permits although perhaps not as lengthy as my last trip report as time is becoming a bit of a luxury. The conference was very interesting, not least of which they predict that SOA is already becoming legacy and among many other topics there is a new buzzword (since 2006, I believe) which is WOA (Web Oriented Architecture). WOA basically refers to the REST method. Gartner indicated that best practice is to adopt WOA when you can instead of SOA. One thing I did get some clarity on is that there is no such thing as a SOA architect! This may be obvious to all of you but in the past year I have repeatedly heard this label used almost as a job position. SOA is simply just a design pattern! The correct terminology is to speak of a solutions architect who employs among other design patterns, the SOA design pattern. We all know that the ESB is also just a design pattern. And now there is discussion that ESBs may actually impede progress within the SOA strategy and it is forecasted that the ESB may also become obsolete within the next two years. Speaking of obsolescence, did you know that some actually even consider AJAX as already becoming obsolete (although it will still be THE major technology for rich internet applications for the next couple of years). Anyway, I have a lot to report -- it was a very productive week as I attended two separate conferences, one on SOA and one on Enterprise Architecture (hint: we need to all learn TOGAF in addition to FEA and DODAF) -- and I will try to provide a summary of some of what I have learned by the end of the week. I have all of the PDFs if anyone is interested. I also ordered the multimedia CD from the SOA conference (not the EA one though as I thought this was all just too much money). By the way, Java and C# are out; Ruby on Rails is now the *hot* language and web application foundation with Python being a close second hot language. (I wish things would slow down a bit so I could get a good grasp *smile*) And hey if anyone can figure out what to do with Second Life, please let me know! I know IBM hosts conferences there as well as a multitude of other companies but I find it very difficult to use.

How Much Will Your SOA Cost?
By: David S. Linthicum, CEO, Linthicum Group
Tuesday, March 20, 2007

I'm consulting now...at the project and strategy levels...and thus finding that a lot of real work needs to be done to get SOAs up-and-running. For most organizations, the first step of their SOA project is to figure out how much this SOA will cost. Thus, you can budget appropriately and obtain the funding.
It’s a good first step, but most organizations that want to build an SOA don't have a clue how to approach the cost estimation process. In many cases, they grossly underestimate the cost of their SOA, hoping their bosses and accountants won't notice later. In other words, go in low to get the approval, and reveal the higher costs later after it's too late...the investment has been made. Not a good management practice, if you ask me, but a pretty common one.
So, how do you calculate the cost of an SOA? While you can't cost out an SOA like a construction project, many of the same notions apply, including: understanding the domain, understanding how much required resources cost, and understanding how the work will get done. Moreover, understand what can go wrong and account for it.
Here are some very “general" guidelines:
Budget to budget. The problem that I'm seeing over and over again is that cost estimates are provided without a clear understanding of the work needed to be done. Indeed, you need to budget some time to create the budget. This means understanding the domain in detail, including:
1. Number of data elements
2. Complexity of data storage technology
3. System complexity
4. Service complexity
5. Process complexity
6. New services needed.
7. Enabling technology
8. Applicable standards
9. Potential risks
Typically it's expressed as:
Cost of SOA = (Cost of Data Complexity + Cost of Service Complexity + Cost of Process Complexity + Enabling Technology Solution)
For instance:
Cost of Data Complexity = (((Number of Data Elements) * Complexity of the Data Storage Technology) * Labor Units))
• Number of Data Elements being the number of semantics you're tracking in your domain, new or derived.
• Complexity of the Data Storage Technology, expressed as a percentage between 0 and 1 (0% to 100%). For instance, Relational is a .3, Object-Oriented is a .6, and ISAM is a .8.
So, at $100 a labor unit, or the amount of money it takes to understand and refine one data element, we could have:
Cost of Data Complexity = (((3,000) * .8) * $100)
Or, Cost of Data Complexity = $150,000 USD Or, the amount of money needed to both understand and refine the data so it fits into your SOA, which is a small part of the overall project by the way.
If you get this, you can get the rest of the cost analysis procedure; just reapply the same notions to:
Cost of Service Complexity
Cost of Process Complexity
Enabling Technology Solution
Some things to remember:
1. This is not really metrics (e.g., function points), because we really don't have historical data to abstract here. In essence, you need to use your own project management and project costing methods; just apply them to this new approach, using the formulas I'm suggesting above.
2. Count on 10 to 20 percent variations in cost for the simple reason that we've not walked down this road before. As we move from project to project, we'll get better at costing out SOA.
3. Make sure you dial in at least 2 major mistakes, meaning, selecting the wrong vendor, or hiring the wrong architect. You may encounter more, but it will almost never be less.
4. Make sure to change cost estimates as scope creep occurs, and it always does. The nice things about using formulas such as the ones I’m expressing here is that, as change occurs, you can quickly see the effect on the budget. Moreover, as change occurs later in the SOA projects, the cost of change goes up exponentially.
Finally, here is a sensible, no-nonsense approach to costing out SOA. While the actual numeric assumptions may be debatable, it's the approach that's refreshing. It would be great to see people start using this model, sharing any data points and providing feedback so it could be refined. Clearly, this would benefit the IT community a lot. I plan to develop a much more sophisticated model in the near future. If you’re interested, let me know, I’ll post it on my Web site.

David Linthicum (Dave) is an internationally known application integration and service oriented architecture expert. In his career Dave has formed many of the ideas behind modern distributed computing including EAI (enterprise application integration), B2B application integration, and service oriented architecture (SOA), approaches and technologies in wide use today.

Currently, Dave is the CEO of the Linthicum Group, LLC, a consulting organization dedicated to excellence in SOA product development, SOA implementation, and corporate SOA strategy. Dave is formerly the CEO of BRIDGEWERX, and is also the former CTO of Mercator Software and has held key technology management roles with a number of organizations including CTO of SAGA Software, Mobil Oil, EDS, AT&T, and Ernst and Young.

Blogging reaching its peak In 2007

Blogging may witness its peak time in the year 2007, with the number of blogs leveling out at 100 million. According to technology predictions by analysts at the research company Gartner, almost 200 million people have already stopped writing their blogs. Daryl Plummer, chief Gartner fellow, said that although some people who make a blog get bored and forget about maintaining it, others love it and are committed to keeping it up. "A lot of people have been in and out of this thing," Plummer said, according to BBC. "Everyone thinks they have something to say, until they're put on stage and asked to say it." Last month the blog-tracking firm Technorati reported that nearly 100,000 new blogs were being created every day, and 1.3 million blog posts were written. The firm, which is keeping track of more than 57 million blogs, says around 55% are "active" and updated at least every three months.

ROI and SOA

This video was also published by TIBCO. Good to see someone has a sense of humor. *smile*

SOA

I just spent a week at two Gartner conferences, which I will blog about later. But I was shown this cute little video: