De video presentaties van deze sessie zijn nu beschikbaar.

Innovatie, Services en Open Source
Rene van Hees, Technical Authority Software bij Thales

http://www.youtube.com/watch?v=2NlzXw37okM

OSGi en Dynamische Services
Marcel Offermans, lid van de Apache Software Foundation

http://www.youtube.com/watch?v=bfEOKva1xhc

Apache Celix en Apache Foundation
Alexander Broekhuis, Software Engineer Luminis, committer Apache Celix

http://www.youtube.com/watch?v=FG04jERMznU

The CHARTER project includes user partners from four key industries: aerospace, automotive, surveillance and medical, each of which develops embedded systems that require high assurance or formal certification in order to meet business or governmental requirements. The four user partners will each validate the CHARTER tools and methodology using industrial applications and actual development scenarios, which will provide feedback for the project and ensure the tools and technologies perform as expected, and deliver the expected improvements in embedded systems development. As part of the evaluation process, metrics will be used to quantify industrial experiences in terms of development effort, cost savings, verification time, etc., to document for others the benefits achieved.

The CHARTER project was established to improve the software development process for developing critical embedded systems. Critical embedded software systems assist, accelerate, and control various aspects of society and are common in cars, aircraft, medical instruments and major industrial and utility plants. These systems are critical to human life and need to be held to the highest standards of performance through formal certification procedures. Improving the quality and robustness of these systems is paramount to their widespread adoption.

Figure: The effort in MDE approach with partial manual coding (adapted from [1])

The generative MDE automates well defined routine activities. An effective metric of depicting economical benefit thereof is effort. The above figure illustrates effort reduction due to automation and reuse in an MDE approach with partial manual coding. Of cause, the generative MDE improves quality as well: error reduction, enforced architecture conformance, and up-to-date documentation are common factors that have positive effect on software quality. But usually these are considered as icing on the cake that is effort reduction. In my experience this perspective on the economical value of MDE is common among both customers and MDE professionals. The perspective can be summarized as “the same with less”.

More with the same

Recently a client tasked me together with its domain experts to assess benefits of applying MDE to a difficult process within the organization. Having analyzed the before and after situations, we came up with estimated economical benefit expressed in effort savings. The estimate was hard to quantify, but “should” have been OK. Although I wanted to share this optimism, I felt that in practice the effort saving would be negligible if not even negative. This paradox was due to the fact that the largest activity in the problem domain was inherently creative and exploratory.

In the figure, the output of a single exploration in this activity is shown as intermediate result, corresponding to line ad. As the figure suggests, code generation directly from the output is not possible (this happens further downstream in development). You may have noticed that the modelling curve rises more steeply towards point a. This rise occurs because modelling requires increased level of domain understanding and more information is needed by semantically rich operations, such as simulation, verification, code generation (eventually), etc. On the other hand, the figure shows effort reduction indicated by distance cd, which is the result of providing end users with proper abstractions, faster access to right knowledge, separation of concerns, DRY modelling, maintained consistency and integrity.

While working efficiency per exploration is likely to  increase (compare ab and cd), the leading concern is quality of the output. Here benefits are early detection of design errors, deep exploration of design choices, better communication and documentation, maximized reuse of domain-specific platforms in further development. Moreover, the domain experts noted that any saved effort would be re-invested in more alternative explorations in search of a more optimal output. This increased number of explorations, is likely to balance out any savings due to higher efficiency.

With these insights, economical benefits were expressed with quality metrics and linked to different business goals than initially thought.

Conclusion

The described MDE assessment targets a highly creative engineering activity that explores alternative choices. In extreme case, the main benefit is not effort reduction, but increased product and process quality. The icing on the cake is that processable models can open opportunities for generative uses as well.

In my experience, such and certainly less extreme quality-driven cases are not exotic. In recent years, quite a few MDE projects I’ve participated in, had benefits strongly linked to quality improvement. What are your MDE experiences with creative activities? What were the economical benefits and how were they conveyed?

References

[1] Thomas Stahl, Markus Voelter, Krzysztof Czarnecki. “Model-Driven Software Development: Technology, Engineering, Management”. Wiley; 1 edition (May 19, 2006)

MDE Approach

Two main choices underpin the process and technology used to develop the interpreter:

1. Execution semantics of the interpreter is defined within the problem domain itself (weighbridge in this case), without translating it to another domain (e.g. .Net or Java) as it is the case with CG. Such definition of semantics is also known as operational semantics. The advantage is reduction of development complexity: out of at least 2 domains needed for CG, only one and the more abstract domain is sufficient.
2. Operational semantics is defined within an MDE framework. This enables customization of modeling language for problem domains besides that of the weighbridge example. Moreover, transformation capabilities are used to define operational semantics.

Figure 1: Domain-specific, nested interpretation (DSNI) MDE framework

Figure 1 shows the MDA framework [1] after it has been adapted to reflect the above mentioned choices. (If you are confused between MDA and MDE, you might find this article useful.) In contrast to MDA, there is no PIM or PSM model, but single computational independent model (CIM) written in DSL. CIM is both source and target of Transformation Tool. Transformation Tool carries out transformation classified as same language, same model. Transformation Definition defines operational semantics. It is not important if Transformation Definition Language (TDL), extends the Metalanguage as in MDA or is customizable by means of meta-specification. Therefore TDL is omitted from the framework and TDL selection criteria are defined instead (see below). Finally, new concept System Context is connected to Transformation Tool. This is due to the fact that interpretation as system exhibits external behavior through communication with other systems.

This approach can be described as nested interpretation, where a domain-specific interpreter is executed (nested) by a generic interpreter. From this perspective, Transformation Tool assumes the role of a generic interpreter and execution of Transformation Definition fills in the role of the domain-specific interpreter.

TDL Selection Criteria

Selection criteria for transformation definition language are:

• declarative modeling
• support for domain-specific notation

These criteria help to reduce development complexity and improve communication with problem domain experts.

Selected MD Technology

AToM3 is a free language workbench written in Python and under development at the Modelling, Simulation and Design Lab (MSDL) in the School of Computer Science of McGill University. The workbench closely matches the DSNI framework and meets the TDL selection criteria.

In AToM3, DSLs and models are described as graphs. From a language specification written in the metalanguage (ER formalism), AToM3 generates a tool to visually manipulate (create and edit) models written in the specified DSL. Model transformations are performed by graph rewriting. The transformations themselves can thus be declaratively expressed as graph-grammar models. However, AToM3 provides no communication infrastructure as needed by the framework.

Proof of Concept

As demonstration, a language specification for the weighbridge domain was defined (see sections domain and weighbridge DSL here) and graph rewriting was used to model operational semantics (see below). Source code of AToM3 itself, being written in Python, was extended to support web services for the communication purpose.

Operational Semantics

As blueprint for operational semantics of the interpreter, we took πDemos [2], a small process-oriented discrete event simulation language. There is a number of πDemos events that change state of a weighbridge system. For each such event, [2] defined the transitions induced on system state. While the original used functional programming language, this work uses graph rewriting and a graph grammar (GG) rule is defined per event.

Table 1: Graph grammar rules of weighbridge events

Table 1 lists the minimum set of required events and their corresponding GG rules. Execution of such rules needs to be globally orchestrated through proper sequencing. The rules, together with execution sequencing, form an operational semantics model of the interpreter.

For complete description of the model, please refer to [3]. In the following, we present a detailed description of an example rule, followed by the execution sequencing model.

Example GG Rule

a) Left-hand side (LHS)

b) Right-hand side (RHS)

Figure 2: Subgraphs of the promoteProcess rule

Rule promoteProcess releases a busy weighbridge (bluish box in Figure 2a) that delays at least one vehicle (yellow box labelled 5). In the new state, the weighbridge remains busy and the blocked vehicle (5) is removed from the head of queue Delay and inserted in waiting queue EL.

The rule is executed if:

This blog is part two of a little example I wrote to test the new threading stuff in .Net 4.0. To read part 1 follow this link.

Short recap from part 1 first. This is what I wrote:
The thing I have to do is get some data from the web in the following pseudo manner:

foreach subject from the list
search for the last 100 tweets on this subject


The part left over was to show some results from the threaded calls from part 1. It so happens to be that, somewhere near the end, I showed the following call as the most nicest and clean threading mechanism:

public static void GetParallel()
{
    Parallel.ForEach(UrlList, url =>
        {
            Get(url);
        }
    );
}

Threadsafe collections

From the example it shows we’re not doing anything useful with the result which is returned by Twitter. A byte array is returned and the thing I want to do is to store the byte array in a collection of byte arrays. Pretty simple huh? Well almost because a potential threading problem is kickin’ in. The standard collections in .Net are not threadsafe. We used to write code with a locking object and lock the piece of code which makes changes to the collection. The new .Net 4.0 has a more smooth solution to this problem: System.Collections.Concurrent.BlockingCollection. And the code above is changed in:

// use a thread safe collection for all threads
BlockingCollection<byte[]> data = new BlockingCollection<byte[]>();
 
Parallel.ForEach(UrlList, url =>
{
    data.Add(Get(url));
}
);
 
// return an ordinary collection (non thread safe) to the outside world
return data.ToList<byte[]>();

So each thread use a thread safe collection to store its results and when we’re ready with all threads the non-threadsafe counterpart is returned because when processing the result we don’t need any multithreaded handling.

Twitter data structure

Twitter returns its data in JSON format. The search API returns the data in the following structure (see also the Twitter API documentation), here is an example:

{
  "results":
  [
    {
      "text":"@twitterapi  http:\/\/tinyurl.com\/ctrefg",
      "to_user_id":396524,
      "to_user":"TwitterAPI",
      "from_user":"jkoum",
       "metadata":
      {
        "result_type":"popular",
        "recent_retweets": 109
      },
      "id":1478555574,
      "from_user_id":1833773,
      "iso_language_code":"nl",
      "source":"<a href="http:\/\/twitter.com\/">twitter<\/a>",
      "profile_image_url":"http:\/\/s3.amazonaws.com\/twitter_production\/profile_images\/118412707\/2522215727_a5f07da155_b_normal.jpg",
      "created_at":"Wed, 08 Apr 2009 19:22:10 +0000"
      },
      ... truncated ...
  ],
  "since_id":0,
  "max_id":1480307926,
  "refresh_url":"?since_id=1480307926&q=%40twitterapi",
  "results_per_page":15,
  "next_page":"?page=2&max_id=1480307926&q=%40twitterapi",
  "completed_in":0.031704,
  "page":1,
  "query":"%40twitterapi"
}

The list of byte arrays we have is a list of bytes in JSON format. Each byte array represents the results of a query and consists of some meta data describing the query and a list with references to all tweets returned by the search query. Each of these byte arrays needs to be transformed in something more meaningful and useful. Here JSON comes to the rescue, it is not only blazingly fast it is also simple. All we have to do is just define the interesting elements in the structure and the serialization will do its work, silently and fast. The structure in a .Net understandable format is:

[DataContract]
public class TwitterSearchResultList
{
    [DataMember]
    public TwitterSearchResult[] results { get; set; }
    [DataMember]
    public string query { get; set; }
    [DataMember]
    public double completed_in { get; set; }
}
 
[DataContract]
public class TwitterSearchResult
{
     [DataMember]
     public string from_user { get; set; }
     [DataMember]
     public string text { get; set; }
     [DataMember]
     public string profile_image_url { get; set; }
}

Transforming one byte array with twitter results is now as easy as:

List<TwitterSearchResultList> tweets = new List<TwitterSearchResultList>();
 
foreach (byte[] tweetlist in allTweetLists)
{
        MemoryStream tweetStream = new MemoryStream(tweetlist);
        DataContractJsonSerializer serializer = 
                new DataContractJsonSerializer(typeof(TwitterSearchResultList));
        TwitterSearchResultList tsrl = 
                (TwitterSearchResultList)serializer.ReadObject(tweetStream);
}

Rendering to html

To render the TwitterSearchReturnList to a presentable format I generated some html. This can be done much cleaner and nicer with XSLT and/or CSS but for the sake of the example I just coded the bare minimum. It looks like:

StringBuilder = new StringBuilder();
stringBuilder.Append("<TABLE>");
foreach (TwitterSearchResult tweet in list.results)
{
     stringBuilder.Append("<TR>");
 
     // the tweeters' avatar
     stringBuilder.AppendFormat("<TD><IMG src=\"{0}\" width=\"48\" heigth=\"48\" /></TD>", tweet.profile_image_url);
     // tweeters' name
     stringBuilder.AppendFormat("<TD>{0}</TD>", tweet.from_user);
     // tweet text; with all urls as an href
     stringBuilder.AppendFormat("<TD>{0}</TD>", Regex.Replace(tweet.text, "(http:/[\\S/]*)", "<a href=\"$1\">$1</A>"));
 
    stringBuilder.Append("</TR>");
}
 
stringBuilder.Append("</TABLE>");

Resulting in the following output:

Source code

The full source code for this example can be downloaded here.
Thank you for reading, questions will be answered, suggestions are more than welcome!
have fun,
florisz

In my opinion it isn’t possible to have these two words in one sentence. I’ve been exposed to thread programming numerous times in the past. Despite the almost heroic nature of being a ‘thread programmer’, I’ve never had the guts to expose this in public. Brag over a few beers about success stories on how well my thread-aware and thread-safe software was running? It has never been part of my vocabulary. One simple reason: it’s way too complicated. Well, at least for me. Even with the aid of thorough design, good examples and extensive mock tests my code remained complicated and incomprehensible. And yes, I tried to refactor, redesign, recode, restructure, whatever, I never could get it right.

Well, up until .Net 4 and its component: System.Threading.Tasks. Included in the very core of the framework. Right out of the box, support for threading for the masses.

Why threading?

It is good to step back for one moment. The 1-million-dollar-question to always ask yourself is: do I really need it? The simple answer is: yes. Because why would you let anyone wait while your machine is sitting there doing nothing? Especially when it is not necessary to wait.

So: yes I need threading to make software as performant as can be to the best of my abilities as a software engineer.

Do I always have to use threading? Of course not. In its most essential form, current CPU’s can do multiple things at the same time (especially when having more cores) and are very fast and therefore often waiting for some other things (like database or web I/O stuff) to end before they can continue. So if you’re in such a situation, and I can assure you, this is most often, threads come to the rescue to really utilize the CPU to its full capacity.

The example

Let’s apply threading to a simple example to show how the new .Net threadinglayer can be utilized to increase performance 40-70% in a very easy manner.

Suppose you want to do a multiple query on Twitter (most popular demo environment) in which you need to return the first 100 hits for a couple of subjects. I have tried this with my webbrowser and you have to do some manual work to return such a list in one piece.

Beforehand I do know getting 100 tweets from Twitter isn’t that difficult, especially with their JSON based REST API (that’s a lot of cool keywords in one sentence!). So one task is easy to implement. I also know that getting something from the internet usually sets my CPU in a comfortable waiting state because network transfer is still way slower than my CPU. So the task is also suitable for multiplexing a couple of times.

Some preparation

First I have to define my testcases. I don’t want to get bogged down into testclasses (although I do agree it would have been better to use them) because hey, this is demo code anyway.

The thing I want to show you is:

In short, I want to show the performance differences between four separate scenarios
-      A non threaded one
-      A classic, pre 4.0, threadpool version
-      A 4.0 version using the new TaskFactory
-      And finally the version using the new Parallel class

These scenarios all do exactly the same thing; only the scheduling of the tasks differs. The thing they have to do is get some data from the web in the following pseudo manner:

foreach subject from the list
search for the last 100 tweets on this subject


The search API in Twitter is pretty straightword. For example, to perform the above for the subject ‘luminis’ the following url will suffice:

http://search.twitter.com/search.json?&q=luminis&rpp=100

To get data from the web I’ve defined a small routine which returns a byte array with the required data. Fortunately WCF made getting data from the web pretty straightforward in .Net, this resulted in the very simple basic routine:

private static byte[] Get(string url)
{
    byte[] data = null;
    WebClient wc = new WebClient();
    try
    {
        data = wc.DownloadData(url);
    }
    catch
    {
        // probable time out, for this demo just quit
    }
    finally
    {
        wc.Dispose();
    }
 
    return data;
}

This routine is part of a class called WebContent. This class also has a public property UrlList which can contain the list of urls for which the content must be read.

private static List _UrlList = new List();
public static List UrlList
{
    get
    {
        return _UrlList;
    }
}

What I have to do now is to iterate over this list and call my simple Get routine for each of the four scenarios.

#1: The non-threaded version

The simple version is the one without any threading at all. We do need this one to baseline our tests so we can do a comparison with the threaded counterparts. The only thing we need to do is read all the urls and the code pretty much resembles the pseudo code shown previously:

public static void GetSequential()
{
    foreach (string url in UrlList)
    {
        Get(url);
    }
}

#2: The 2.0 ThreadPool version

Actually I didn’t want to make the ThreadPool version because as said earlier I don’t get this ‘old’ thread programming very well. But as a reference I’ve hacked a version for which I can’t take any liabilities. It is just here for reference purposes. Note that I need two routines to achieve the same goal. Additionally I’ve changed the thread of the main to [MTAThreadAttribute] otherwise you will receive to message:

Anyway the code looks like:

public static void GetWithThreadPool()
{
    _ResetEvents = new ManualResetEvent[UrlList.Count];
 
    Random rand = new Random();
    for (int i = 0; i < UrlList.Count; i++)
    {
        _ResetEvents[i] = new ManualResetEvent(false);
        ThreadPool.QueueUserWorkItem(new WaitCallback(DoThreadPoolWork), (object)i);
    }
 
    WaitHandle.WaitAll(_ResetEvents);
}
 
private static void DoThreadPoolWork(object o)
{
    int index = (int)o;
 
    Get(UrlList[index]);
 
    _ResetEvents[index].Set();
}

Pretty impressive but incomprehensible, huh?
It doesn’t feel right that you have to add all these complex things just to get a bit of multi-threading. To me, the only easy part is the Get call somewhere near the end. All the other stuff is thread overhead. And if you would need more that 25 threads this code won’t work. But let’s not dwell on these passed issues and move over to the 4.0 versions.

#3: The  4.0 TaskFactory version

The task factory shows what the thread overhead could be if the framework is more powerful than the 2.0 version. Still it is some overhead but the code looks more like the pseudo code than the previous implementation:

public static void GetWithTaskFactory()
{
    Task[] tasks = new Task[UrlList.Count];
    int counter = 0;
 
    foreach (string url in UrlList)
    {
        var task = Task.Factory.StartNew(() =>
                                 Get(url), TaskCreationOptions.LongRunning);
        tasks[counter] = task;
        counter++;
    }
 
    Task.WaitAll(tasks);
}

The thing I need to do is to create a task for each subject and, with a lambda expression, call my good old private Get() on this task. Finally I just have to wait until all tasks are finished.

#4: The 4.0 Parallel version

The ultimate version is left to the end. The new Parallel class makes it possible to write threaded code without almost no (thread) overhead. It looks like:

public static void GetParallel()
{
    Parallel.ForEach(UrlList, url =>
        {
            Get(url);
        }
    );
}

Isn’t that just beautiful? It also resembles the pseudo code pretty well. Okay you have to understand the lambda expression with the anonymous delegate syntax and semantics but if you do it is just an ordinary for-loop. As far as I’m concerned: this is as it should be.

Performance results

So far the differences in the source code of the four implementations. To start each of the separate scenarios a little bit of timing code is needed to calculate the total of milliseconds the code runs.

private delegate void GetContentDelegate();
 
private static string DoTimedOperation(GetContentDelegate getForumContentDelegate)
{
    DateTime start = DateTime.Now;
 
    // perform the actual operation
    getForumContentDelegate();
 
    DateTime end = DateTime.Now;
    TimeSpan elapsed = end - start;
 
    return elapsed.TotalMilliseconds.ToString() + "ms";
}

And with the aid of a delegate the timing function is called four times as in:

private void StartSequential_Click(object sender, EventArgs e)
{
    this.lblSequentialTime.Text =
        DoTimedOperation(new GetContentDelegate(WebContent.GetSequential));
}
 
private void StartThreadPool_Click(object sender, EventArgs e)
{
    this.lblThreadPoolTime.Text =
        DoTimedOperation(new GetContentDelegate(WebContent.GetWithThreadPool));
}
 
private void StartTaskFactoryl_Click(object sender, EventArgs e)
{
    this.lblTaskFactoryTime.Text =
        DoTimedOperation(new GetContentDelegate(WebContent.GetWithTaskFactory));
}
 
private void StartParallel_Click(object sender, EventArgs e)
{
    this.lblParallelTime.Text =
        DoTimedOperation(new GetContentDelegate(WebContent.GetParallel));
}

I ran a few tests on my machine and one of them resulted in the following numbers: (By the way my machine is a Macbook Pro running Windows 7 in a Virtual Box with 1 processor and 2048 Mb memory.)

Of course you have to try this on your own machine but it is safe to say that the performance increase is worth the overhead in thread programming.

I have analyzed the thread behavior with the Performance Monitor (PerfMon). In the analysis I was interested in:
- # of processor threads (blue)
-      # of logical threads for the process (green)
-      # of physical threads for the process (red)
These are the results of one test I have run:

You can see the framework too needs some threads when the non threaded version is started. When the threadpool scenario starts you clearly see the increase in threads. The threadpool is managed by .Net so this amount doesn’t decrease when the scenario is finished. This does happen with the third scenario. Here the threads are in control of the routine and you can clearly see that the threads are released (at least to the system, the framework has a different view on it). In the fourth scenario again the threads are fully controlled by the framework.
Interesting to see is: in both the threadpool and the parallel scenario only a limited amount of threads are created (approximately 5 or 6) while in the Taskfactory 12 threads are created (for each subject one). It is hard to say what exactly happens behind the scenes. Do all logical threads have a physical thread associated? And do these threads run ’simultaneously’ or not? I have to say I don’t know.
Then there is one thing I don’t understand at all. How is it possible (near the end) that the total number of physical threads of the process is higher than the system amount of physical threads?! If somebody can tell me this I would be very hapy!

Conclusion

In a next blog I will extend this little example. So far I have concentrated on threads, starting and synchronizing them to show the performance differences. I have experimental proof it is worth taking an effort to learn more on the new .Net 4.0 threading stuff.
The part that is left out is the handling of the data and the presentation of the results.
This part is described in part 2 and can be found here.

have fun, florisz

Download the zip for the full source code here.

References

Twitter Search API

MSDN on System.Threading.Tasks

.Net 4.0 and System.Threading.Tasks

Threadpool example

Task Parallelism in C# 4.0 with System.Threading.Tasks

Big Data, the cloud

One of the main trends I noticed is the interest in Big Data (mainly Apache Cassandra) and Big Data processing in various shapes and forms (e.g. Apache Lucene, Apache Hadoop). In relation to many of these, we find a ring of ‘cloudness’: the products tend to allow distribution and replication of data and functionality.

But it doesn’t stop there. It’s no surprise to find cloud-references in talks about Apache Tuscany (including a talk on Building cloud native software, which I regrettably missed), but for instance Tomcat is making its move into cloud territory with Stratos.

OSGi all around

Of course I have a vested interest in OSGi, and my talk during the OSGi track on friday shows this. However, apart from in its own track, OSGi and OSGi-based technologies popped up in a number of other tracks. To name a few,

• Apache Sling is an OSGi based web application platform, and showed up in the Content Technologies track,
• Apache Geronimo 3.0 (in the Content Technologies track) is now based on OSGi, and
• in the Enterprise Track, a number of sessions were decidated to Apache Aries, which focuses on the OSGi Enterprise specification.

It is an interesting development that OSGi is now mainly being referenced in the web- and enterprise spaces, whereas it started out as a specification for embedded devices.

Does this mean that OSGi is really getting traction in the software community? Yes and no. I believe the thing that is really getting traction is the notions of modularity on the one hand, and µServices on the other. OSGi is currently the main technology that marries those two notions.

Business is not a dirty word

I noticed there were roughly three kinds of talks,

• Community talks are all about how Apache works, and how open source software fits into the world around it,
• Technology talks focus on some Apache project, or a combination of projects, and go into the technical details, and
• Industry talks that show how the projects are used in industry.

The last kind of talk shows how industry, the ‘people with the problem’ use open source technology to run their business. No, I probably don’t really care about the products and services you deliver, but I am very interested in your case studies in using open source in your daily life.

A few days ago, Slashdot ran a story Paid Developers Power the Linux Kernel. In the light of sessions on open source in industry (I counted about half a dozen on this year’s schedule), this is not that surprising: if you’re a good enough developer to become a committer in an open source project, it’s very unlikely you’re working as a janitor during the daytime. All of us work for a corporation in some way, and many of us work on open source project during our paid time. I believe this is a good thing!

• MetaEdit+ Modeler provides customizable DSM functionality for multiple users, multiple projects, running on all major platforms.
• MetaEdit+ Workbench i) allows building custom modeling languages (DSLs), and text generators and 2) includes the functionality of MetaEdit+ Modeler and MetaEdit+ API (the latter is not reviewed in this document).

This review is written from the MDE perspective and will cover major MDE functionally related to specification of modeling languages. For a complete picture of MetaEdit+, readers are advised to consider other aspects (e.g. collaboration, versioning, etc…) as well.

This review covers MetaEdit+ Workbench version 4.5.

Language Specification

MetaEdit+ supports graph-like visual languages represented as diagrams, matrixes or tables. There is a limited support for spatial languages: touch and containment relationships are derived from canvas coordinates of modeling elements. There is no actual tool support to preserve these relationships: for example, as a modeller moves a “container” element, contained elements do not move along as expected, but remain at old coordinates.

In MetaEdit+, languages are specified with a set of specialized tools. In the following, we describe the tools per each aspect of the visual language definition: abstract syntax, concrete syntax, static and dynamic semantics.

Abstract Syntax

This aspect is defined with GOPPRR metatypes. GOPPRR is an acronym for metatypes Graph, Object, Property, Port, Role and Relationship. For each metatype, there is a form-based tool, e.g. Object tool allows specification of object types  and Graph tool allows assembling types produced with the other tools into a specification of abstract syntax. GOPPRR tools support single inheritance.

Graph tool also allows linking DSL objects to graphs of other DSLs through decomposition and explosion structures. Furthermore, through sharing language concepts (of any OPPRR metatype) among graphs, DSLs can be integrated so that changes in one model can be automatically reflected in models based on different languages.

An alternative to these form-based tools for abstract syntax specification is a visual metamodeling DSL. However, this functionality is best used as easy start-up leading to automated generation of barebone GOPPRR metamodels. Once a language developer changes a GOPPRR metamodel (which is inevitable), visual metamodeling is best discontinued to avoid manual round-trip between the two metamodels.

Concrete Syntax

By default, MetaEdit+ provides generic symbols. However, language developers are free to specify custom symbols for objects, roles and relationships. These symbols are either defined with a WYSIWYG vector drawing tool or imported from vector graphics (SVG) or bitmap files. Symbols can display text, property values and dynamic outputs produced by text generators (more on generators in section M2T Transformation). Moreover, symbols or their parts can be conditionally displayed. Finally, symbols can be reused among different DSLs via a symbol library.

MetaEdit+ does not directly support multiple concrete syntaxes per language, which (the lack of such support) is still a common practice among language workbenches. However, its capability to display symbols based on conditions allows to work around this limitation.

Static Semantics

This aspect covers constraints and business rules. The purpose of these rules is to ensure a consistent and valid model.

In general, DSM tools should verify a model against the static semantics of its DSL at different times. These times can be classified as ‘live’ (i.e. when a user is modelling) and ‘batch’ (i.e. invoked on events caused by actions such as user demand, model saving or transformation). Furthermore, tool actions following violation of a constraint can be classified as prevention (i.e. a violating action is canceled and a warning message is displayed) or merely informative (i.e. a violating action is allowed, but model will display clues about invalid constructions until the effect of the action is corrected).

MetaEdit’s Constraint tool (available from the Graph tool) allows ‘live’ checks against constraints and preventive protection of models (‘live’ and ‘preventive’ in the terms of the above classifications). The tool is very expressive and easy to use, but covers only limited number of types of constraints, namely:

• object connectivity in a relationship
• object occurrence in a model
• ports involved in a relationship
• property uniqueness

More advanced constraints have to rely on MERL generator (see section M2T Transformation), which can inform users about invalid constructions during modeling (‘live’ and ‘informative’ in the terms of the above classifications). MERL generator can also be used for ‘batch informative’ and ‘batch preventive’ checks: a checking report can be run on demand or included as preventive check before any other transformation is carried out.

Dynamic Semantics

MetaEdit+ can define dynamic semantics through a process of translating DSL concepts to concepts in another target domain with defined dynamic semantics. Examples of target domains in code generation applications are e.g. C++ or Java. A major benefit of language workbenches is that they are capable of automating this and other useful kinds of processes.

PROCESS AUTOMATION

MDE applications need capabilities to automate processes in which models are inputs and outputs. MetaEdit+ provides various levels of support for model-to-model (M2M), model-to-text (M2T) (e.g. in code generation applications) and text-to-model (T2M) (import of legacy code, data type definitions, etc. into models) types of transformations. (The latter transformation type is not reviewed.)

M2T Transformation

Text (and more specifically code) generation is accomplished with Generator tool that can efficiently navigate models, filter and access information, and output text into external files, Generator Output tool and DSL symbols.  All these tasks are specified with imperative language MERL. While MERL is very concise and efficient for most of these tasks, I think that navigation and access tasks are better accomplished in a declarative way.

MERL generators are defined per graph type (i.e. per DSL) and can be acquired from supertypes of a given graph type via an inheritance hierarchy. If a generator has to be used for different graph types, then the generator should be defined for the common parent graph type. On the other hand, DSL developer can define new or redefine generators already provided by parent graph types.

Finally, MERL provides support for modularization by allowing includes of generators in other generators. Making modular generators pays off well, as there are many reuse opportunities in MetaEdit+: generators can be reused not only for text generation but also in concrete syntax (symbols) and validation/reporting purposes (symbols, generator output tool).

M2M Transformation

Models can be transformed 1) programmatically via the SOAP and WebServices-based API of MetaEdit+ (this option requires product component MetaEdit+ API) or 2) through code generation of an intermediate external representation (in the XML format) and consequent import thereof as new model.

These two options amount to a generic support at a minimum level that is commonly provided nearly by all language workbenches. Moreover, code generation of an intermediate representation cannot implement in-place M2M transformations, of which application examples are: model optimization, model layout, model interpretation, model weaving and any repeatable model manipulation in general.

OTHER

• DSL evolution: MetaEdit+ updates existing models instantly yet non-destructively to reflect changes in DSLs.  The update policy ensures that models created with the older DSL versions are not broken and remain usable with existing generators. Instant update is also very useful when fine-tuning a DSL with end users.
• According to MetaCase, a MetaEdit+ project can hold over 4 billion objects. A typical project would contain about 40-100 models (graphs).
• In the multi-user version, users can simultaneously access and share all models within a Repository. Locking is made at the object-level, so several users could collaboratively work on the same model at the same time.
• Multi-user collaboration in MetaEdit+, product line analysis of commonality and variability and proper separation of concerns reduce the need for version control as it is known in software engineering. Therefore MetaEdit+ does not provide its own versioning system. Best practices for versioning with MetaEdit+ can be found here.
• Model interoperability: by default, all models and DSLs can be exported in an XML format. The schemas are very simple, which make it easy to post-process such files if needed. Moreover, the M2T transformation capabilities of MetaEdit+ enable DSL developers to easily create custom export generators.

CONCLUSION

MetaEdit+ is a versatile language workbench that enables building high quality visual DSLs for any kind of domain, be it technical or business. Another key quality of MetaEdit+ is efficient DSL/GOPPRR tools, which allow light-weight, agile and fast DSL development and evolution. A testament to this quality is the fact that MetaCase is one of few language workbench makers that routinely designs and builds DSLs in improvisation with audience at conferences, workshops, etc. In my opinion, this impressive productivity is possible because GOPPRR tools are based on paradigms that are optimum for DSL development (DSM for DSM so to speak).

Highlights of MetaEdit+ are:

• Proper level of abstraction: DSL developers are completely shielded from details of how DSM-tools are implemented. DSL development tools focus on essential abstractions for specification of languages and generators.
• High-levels of automation: DSM-tools are completely and automatically generated from abstract language specifications.
• High quality of tools: each DSL development task has its own dedicated tool.
• Numerous enhancements: high integration of tools, non-destructive evolution of languages, inheritance mechanism, reuse opportunities for types, symbols and generators, visual metamodeling, etc.
• Very cheap introductory license.

Naturally, there are a few drawbacks as well:

• No specific support for model-to-model transformation.
• Somewhat limited constraints support.
• Limited support for spatial relations.
• Uncommon user interface.
• Form-based GOPPRR tools prevent a global overview of a metamodel.
• Expensive  standard licenses.

Code generation applications are the oldest tradition in MDE and this is where MetaEdit+ excels. As MDE discovers new applications, my experience is that the code generation specialization becomes restrictive. Admittedly, it is possible to implement some types of M2M transformations with code generation (via intermediate representation). However, the problem with this workaround is that it introduces accidental complexity both to MDE developers and more importantly to end users (that have to keep repeating the generate/import routine, sometimes complicated by model merge).

That said, in my opinion MetaEdit+ gets the big things right. Whether its shortcomings are little things is a subjective matter that is best evaluated in the context of a concrete problem domain.

De laatste tijd heb ik een lijstje gemaakt met uitspraken die volgens mij vrijwel nooit buiten kantoormuren geuit worden:

• doorakkeren
• inschieten
• oppakken
• opschalen
• verwachting neerleggen
• doorpakken
• aanvliegen
• ergens een ei over leggen
• issue ergens beleggen
• ergens op acteren
• iets platslaan
• aan de pruttel helpen
• de hele spullenboel
• doorsnijding
• propositie

Wie heeft leuke toevoegingen?

PhoneGap is een interessante open source alternatief voor het schrijven van native applicaties voor elk (mobiel) platform dat er is. In het kort komt het erop neer dat PhoneGap zorgt dat je een HTML applicatie met javascript.  De specifiek API, zoals location, contact, e.d. worden afgeschermd door een standaard API van PhoneGap.

iPhone

Voor de iPhone kan de HTML applicatie gewoon worden aangeboden via de appstore. Dit is dan ook direct de truc waardoor er voldoende rechten zijn om de hardware aan te spreken. Er zijn al vele applicatie geplaatst in de appstore (zie een selectie in www.phonegap.com/apps).  Er is tevens een getting started en er zijn  extra plugins beschikbaar

Ondersteunde platformen

Naast iPhone wordt zowel Android, Blackberry, Symbian, Palm, N900 en Windows Mobile. Ook Windows Mobile 7 is zodra dit uitkomt eenvoudig te ondersteunen en ze zullen ook geen blokkade opwerpen in het voordeel van silverlight. Interesante gedachte is natuurlijk ook de iets minder mobiele system met Linux, Windows MacOS hebben ook allemaal een browser.

Voor meer informatie zie www.phonegap.com