This blog is part two of a little example I wrote to test the new threading stuff in .Net 4.0. To read part 1 follow this link.

Short recap from part 1 first. This is what I wrote:
The thing I have to do is get some data from the web in the following pseudo manner:

foreach subject from the list
search for the last 100 tweets on this subject


The part left over was to show some results from the threaded calls from part 1. It so happens to be that, somewhere near the end, I showed the following call as the most nicest and clean threading mechanism:

public static void GetParallel()
{
    Parallel.ForEach(UrlList, url =>
        {
            Get(url);
        }
    );
}

Threadsafe collections

From the example it shows we’re not doing anything useful with the result which is returned by Twitter. A byte array is returned and the thing I want to do is to store the byte array in a collection of byte arrays. Pretty simple huh? Well almost because a potential threading problem is kickin’ in. The standard collections in .Net are not threadsafe. We used to write code with a locking object and lock the piece of code which makes changes to the collection. The new .Net 4.0 has a more smooth solution to this problem: System.Collections.Concurrent.BlockingCollection. And the code above is changed in:

// use a thread safe collection for all threads
BlockingCollection<byte[]> data = new BlockingCollection<byte[]>();
 
Parallel.ForEach(UrlList, url =>
{
    data.Add(Get(url));
}
);
 
// return an ordinary collection (non thread safe) to the outside world
return data.ToList<byte[]>();

So each thread use a thread safe collection to store its results and when we’re ready with all threads the non-threadsafe counterpart is returned because when processing the result we don’t need any multithreaded handling.

Twitter data structure

Twitter returns its data in JSON format. The search API returns the data in the following structure (see also the Twitter API documentation), here is an example:

{
  "results":
  [
    {
      "text":"@twitterapi  http:\/\/tinyurl.com\/ctrefg",
      "to_user_id":396524,
      "to_user":"TwitterAPI",
      "from_user":"jkoum",
       "metadata":
      {
        "result_type":"popular",
        "recent_retweets": 109
      },
      "id":1478555574,
      "from_user_id":1833773,
      "iso_language_code":"nl",
      "source":"<a href="http:\/\/twitter.com\/">twitter<\/a>",
      "profile_image_url":"http:\/\/s3.amazonaws.com\/twitter_production\/profile_images\/118412707\/2522215727_a5f07da155_b_normal.jpg",
      "created_at":"Wed, 08 Apr 2009 19:22:10 +0000"
      },
      ... truncated ...
  ],
  "since_id":0,
  "max_id":1480307926,
  "refresh_url":"?since_id=1480307926&q=%40twitterapi",
  "results_per_page":15,
  "next_page":"?page=2&max_id=1480307926&q=%40twitterapi",
  "completed_in":0.031704,
  "page":1,
  "query":"%40twitterapi"
}

The list of byte arrays we have is a list of bytes in JSON format. Each byte array represents the results of a query and consists of some meta data describing the query and a list with references to all tweets returned by the search query. Each of these byte arrays needs to be transformed in something more meaningful and useful. Here JSON comes to the rescue, it is not only blazingly fast it is also simple. All we have to do is just define the interesting elements in the structure and the serialization will do its work, silently and fast. The structure in a .Net understandable format is:

[DataContract]
public class TwitterSearchResultList
{
    [DataMember]
    public TwitterSearchResult[] results { get; set; }
    [DataMember]
    public string query { get; set; }
    [DataMember]
    public double completed_in { get; set; }
}
 
[DataContract]
public class TwitterSearchResult
{
     [DataMember]
     public string from_user { get; set; }
     [DataMember]
     public string text { get; set; }
     [DataMember]
     public string profile_image_url { get; set; }
}

Transforming one byte array with twitter results is now as easy as:

List<TwitterSearchResultList> tweets = new List<TwitterSearchResultList>();
 
foreach (byte[] tweetlist in allTweetLists)
{
        MemoryStream tweetStream = new MemoryStream(tweetlist);
        DataContractJsonSerializer serializer = 
                new DataContractJsonSerializer(typeof(TwitterSearchResultList));
        TwitterSearchResultList tsrl = 
                (TwitterSearchResultList)serializer.ReadObject(tweetStream);
}

Rendering to html

To render the TwitterSearchReturnList to a presentable format I generated some html. This can be done much cleaner and nicer with XSLT and/or CSS but for the sake of the example I just coded the bare minimum. It looks like:

StringBuilder = new StringBuilder();
stringBuilder.Append("<TABLE>");
foreach (TwitterSearchResult tweet in list.results)
{
     stringBuilder.Append("<TR>");
 
     // the tweeters' avatar
     stringBuilder.AppendFormat("<TD><IMG src=\"{0}\" width=\"48\" heigth=\"48\" /></TD>", tweet.profile_image_url);
     // tweeters' name
     stringBuilder.AppendFormat("<TD>{0}</TD>", tweet.from_user);
     // tweet text; with all urls as an href
     stringBuilder.AppendFormat("<TD>{0}</TD>", Regex.Replace(tweet.text, "(http:/[\\S/]*)", "<a href=\"$1\">$1</A>"));
 
    stringBuilder.Append("</TR>");
}
 
stringBuilder.Append("</TABLE>");

Resulting in the following output:

Source code

The full source code for this example can be downloaded here.
Thank you for reading, questions will be answered, suggestions are more than welcome!
have fun,
florisz

In my opinion it isn’t possible to have these two words in one sentence. I’ve been exposed to thread programming numerous times in the past. Despite the almost heroic nature of being a ‘thread programmer’, I’ve never had the guts to expose this in public. Brag over a few beers about success stories on how well my thread-aware and thread-safe software was running? It has never been part of my vocabulary. One simple reason: it’s way too complicated. Well, at least for me. Even with the aid of thorough design, good examples and extensive mock tests my code remained complicated and incomprehensible. And yes, I tried to refactor, redesign, recode, restructure, whatever, I never could get it right.

Well, up until .Net 4 and its component: System.Threading.Tasks. Included in the very core of the framework. Right out of the box, support for threading for the masses.

Why threading?

It is good to step back for one moment. The 1-million-dollar-question to always ask yourself is: do I really need it? The simple answer is: yes. Because why would you let anyone wait while your machine is sitting there doing nothing? Especially when it is not necessary to wait.

So: yes I need threading to make software as performant as can be to the best of my abilities as a software engineer.

Do I always have to use threading? Of course not. In its most essential form, current CPU’s can do multiple things at the same time (especially when having more cores) and are very fast and therefore often waiting for some other things (like database or web I/O stuff) to end before they can continue. So if you’re in such a situation, and I can assure you, this is most often, threads come to the rescue to really utilize the CPU to its full capacity.

The example

Let’s apply threading to a simple example to show how the new .Net threadinglayer can be utilized to increase performance 40-70% in a very easy manner.

Suppose you want to do a multiple query on Twitter (most popular demo environment) in which you need to return the first 100 hits for a couple of subjects. I have tried this with my webbrowser and you have to do some manual work to return such a list in one piece.

Beforehand I do know getting 100 tweets from Twitter isn’t that difficult, especially with their JSON based REST API (that’s a lot of cool keywords in one sentence!). So one task is easy to implement. I also know that getting something from the internet usually sets my CPU in a comfortable waiting state because network transfer is still way slower than my CPU. So the task is also suitable for multiplexing a couple of times.

Some preparation

First I have to define my testcases. I don’t want to get bogged down into testclasses (although I do agree it would have been better to use them) because hey, this is demo code anyway.

The thing I want to show you is:

In short, I want to show the performance differences between four separate scenarios
-      A non threaded one
-      A classic, pre 4.0, threadpool version
-      A 4.0 version using the new TaskFactory
-      And finally the version using the new Parallel class

These scenarios all do exactly the same thing; only the scheduling of the tasks differs. The thing they have to do is get some data from the web in the following pseudo manner:

foreach subject from the list
search for the last 100 tweets on this subject


The search API in Twitter is pretty straightword. For example, to perform the above for the subject ‘luminis’ the following url will suffice:

http://search.twitter.com/search.json?&q=luminis&rpp=100

To get data from the web I’ve defined a small routine which returns a byte array with the required data. Fortunately WCF made getting data from the web pretty straightforward in .Net, this resulted in the very simple basic routine:

private static byte[] Get(string url)
{
    byte[] data = null;
    WebClient wc = new WebClient();
    try
    {
        data = wc.DownloadData(url);
    }
    catch
    {
        // probable time out, for this demo just quit
    }
    finally
    {
        wc.Dispose();
    }
 
    return data;
}

This routine is part of a class called WebContent. This class also has a public property UrlList which can contain the list of urls for which the content must be read.

private static List _UrlList = new List();
public static List UrlList
{
    get
    {
        return _UrlList;
    }
}

What I have to do now is to iterate over this list and call my simple Get routine for each of the four scenarios.

#1: The non-threaded version

The simple version is the one without any threading at all. We do need this one to baseline our tests so we can do a comparison with the threaded counterparts. The only thing we need to do is read all the urls and the code pretty much resembles the pseudo code shown previously:

public static void GetSequential()
{
    foreach (string url in UrlList)
    {
        Get(url);
    }
}

#2: The 2.0 ThreadPool version

Actually I didn’t want to make the ThreadPool version because as said earlier I don’t get this ‘old’ thread programming very well. But as a reference I’ve hacked a version for which I can’t take any liabilities. It is just here for reference purposes. Note that I need two routines to achieve the same goal. Additionally I’ve changed the thread of the main to [MTAThreadAttribute] otherwise you will receive to message:

Anyway the code looks like:

public static void GetWithThreadPool()
{
    _ResetEvents = new ManualResetEvent[UrlList.Count];
 
    Random rand = new Random();
    for (int i = 0; i < UrlList.Count; i++)
    {
        _ResetEvents[i] = new ManualResetEvent(false);
        ThreadPool.QueueUserWorkItem(new WaitCallback(DoThreadPoolWork), (object)i);
    }
 
    WaitHandle.WaitAll(_ResetEvents);
}
 
private static void DoThreadPoolWork(object o)
{
    int index = (int)o;
 
    Get(UrlList[index]);
 
    _ResetEvents[index].Set();
}

Pretty impressive but incomprehensible, huh?
It doesn’t feel right that you have to add all these complex things just to get a bit of multi-threading. To me, the only easy part is the Get call somewhere near the end. All the other stuff is thread overhead. And if you would need more that 25 threads this code won’t work. But let’s not dwell on these passed issues and move over to the 4.0 versions.

#3: The  4.0 TaskFactory version

The task factory shows what the thread overhead could be if the framework is more powerful than the 2.0 version. Still it is some overhead but the code looks more like the pseudo code than the previous implementation:

public static void GetWithTaskFactory()
{
    Task[] tasks = new Task[UrlList.Count];
    int counter = 0;
 
    foreach (string url in UrlList)
    {
        var task = Task.Factory.StartNew(() =>
                                 Get(url), TaskCreationOptions.LongRunning);
        tasks[counter] = task;
        counter++;
    }
 
    Task.WaitAll(tasks);
}

The thing I need to do is to create a task for each subject and, with a lambda expression, call my good old private Get() on this task. Finally I just have to wait until all tasks are finished.

#4: The 4.0 Parallel version

The ultimate version is left to the end. The new Parallel class makes it possible to write threaded code without almost no (thread) overhead. It looks like:

public static void GetParallel()
{
    Parallel.ForEach(UrlList, url =>
        {
            Get(url);
        }
    );
}

Isn’t that just beautiful? It also resembles the pseudo code pretty well. Okay you have to understand the lambda expression with the anonymous delegate syntax and semantics but if you do it is just an ordinary for-loop. As far as I’m concerned: this is as it should be.

Performance results

So far the differences in the source code of the four implementations. To start each of the separate scenarios a little bit of timing code is needed to calculate the total of milliseconds the code runs.

private delegate void GetContentDelegate();
 
private static string DoTimedOperation(GetContentDelegate getForumContentDelegate)
{
    DateTime start = DateTime.Now;
 
    // perform the actual operation
    getForumContentDelegate();
 
    DateTime end = DateTime.Now;
    TimeSpan elapsed = end - start;
 
    return elapsed.TotalMilliseconds.ToString() + "ms";
}

And with the aid of a delegate the timing function is called four times as in:

private void StartSequential_Click(object sender, EventArgs e)
{
    this.lblSequentialTime.Text =
        DoTimedOperation(new GetContentDelegate(WebContent.GetSequential));
}
 
private void StartThreadPool_Click(object sender, EventArgs e)
{
    this.lblThreadPoolTime.Text =
        DoTimedOperation(new GetContentDelegate(WebContent.GetWithThreadPool));
}
 
private void StartTaskFactoryl_Click(object sender, EventArgs e)
{
    this.lblTaskFactoryTime.Text =
        DoTimedOperation(new GetContentDelegate(WebContent.GetWithTaskFactory));
}
 
private void StartParallel_Click(object sender, EventArgs e)
{
    this.lblParallelTime.Text =
        DoTimedOperation(new GetContentDelegate(WebContent.GetParallel));
}

I ran a few tests on my machine and one of them resulted in the following numbers: (By the way my machine is a Macbook Pro running Windows 7 in a Virtual Box with 1 processor and 2048 Mb memory.)

Of course you have to try this on your own machine but it is safe to say that the performance increase is worth the overhead in thread programming.

I have analyzed the thread behavior with the Performance Monitor (PerfMon). In the analysis I was interested in:
- # of processor threads (blue)
-      # of logical threads for the process (green)
-      # of physical threads for the process (red)
These are the results of one test I have run:

You can see the framework too needs some threads when the non threaded version is started. When the threadpool scenario starts you clearly see the increase in threads. The threadpool is managed by .Net so this amount doesn’t decrease when the scenario is finished. This does happen with the third scenario. Here the threads are in control of the routine and you can clearly see that the threads are released (at least to the system, the framework has a different view on it). In the fourth scenario again the threads are fully controlled by the framework.
Interesting to see is: in both the threadpool and the parallel scenario only a limited amount of threads are created (approximately 5 or 6) while in the Taskfactory 12 threads are created (for each subject one). It is hard to say what exactly happens behind the scenes. Do all logical threads have a physical thread associated? And do these threads run ’simultaneously’ or not? I have to say I don’t know.
Then there is one thing I don’t understand at all. How is it possible (near the end) that the total number of physical threads of the process is higher than the system amount of physical threads?! If somebody can tell me this I would be very hapy!

Conclusion

In a next blog I will extend this little example. So far I have concentrated on threads, starting and synchronizing them to show the performance differences. I have experimental proof it is worth taking an effort to learn more on the new .Net 4.0 threading stuff.
The part that is left out is the handling of the data and the presentation of the results.
This part is described in part 2 and can be found here.

have fun, florisz

Download the zip for the full source code here.

References

Twitter Search API

MSDN on System.Threading.Tasks

.Net 4.0 and System.Threading.Tasks

Threadpool example

Task Parallelism in C# 4.0 with System.Threading.Tasks

I was looking for a simple solution for paging in a ASP.MVC2 project. Though we already use MVCContrib.Grid this was the first place I searched.

The solution MVCContrib offers is elegant in the way that they separated the concept of paging, a ui element to show paging (next , previous, …) and a grid to show the content. This separation allows us to using paging on a custom table based page as well.

Because we are using ViewModels a little extra effort was needed to retain the paging information from the domain layer in the view layer.

Step by step:

* Get a Querable from repository using Domain Objects

IQueryable<DomainObject> domainObjects = rep.GetAll()

* Filter and Sort using Linq

domainObjects = domainObjects.Where(...).OrderBy(...)

* Get the data

IPagination<DomainObject> pageOfData domainObjects.AsPagination( page, pageSize)

* Map to the ViewModel

IPagination<DomainObjectViewModel> model = DomainObjectViewModel.Map( pageOfData )

* Mapping needs to retain the page information from the domain

var list = new List<DomainObjectViewModel>();
foreach (var domainObject in domainObjects)
{
list.Add(Map(domainObject));
}
new CustomPagination<DomainObjectViewModel>(list.AsEnumerable(),
    pageOfData.PageNumber, 
    pageOfData.PageSize, 
    pageOfData.TotalItems);

* Show the information on a page in a grid

<%= Html.Grid(Model).AutoGenerateColumns() %>

* Show the pager

<%= Html.Pager(Model).First( "<<").Last(">>").Next(">").Previous("<")
.Format( "Item {0} - {1} van {2} ") %>

Dates in XSD

When you write an XSD you have (at least) 3 options to use dates and/or times:

• xsd:date
• xsd:dateTime
• xsd:time

Most of the time you probably choose the “dateTime”, since that maps nicely to the DateTime type in .NET. But you might choose “date” for a birthday for example, and you might even choose “time” for uh, well for a time of some sort…

For his post I’ll use the following XSD. It’s a bit contrived but I like this better than “MyDate” and “ADateTimeField”:

<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <xsd:element name="Book" type="BookType"></xsd:element>
 
  <xsd:complexType name="BookType">
    <xsd:sequence>
      <xsd:element name="Title" type="xsd:string"/>
      <xsd:element name="Price" type="xsd:float" nillable="true"/>
      <xsd:element name="PrintedAt" type="xsd:date"/>
      <xsd:element name="SoldAt" type="xsd:dateTime"/>
      <xsd:element name="WillReadTodayAt" type="xsd:time"/>
    </xsd:sequence>
  </xsd:complexType>
</xsd:schema>

Generate classes from your XSD

Both Mono and Microsoft have command-line tools that generate classes from your XSD. Very convenient and a good thing to do when you need simple entities. We did this in a project where we had both C# and Java code. On both sides we generated the classes from the same XSD.

The syntax of both tools is more or less the same:

xsd book.xsd /classes /namespace:BookStore.Books

will work on both systems.
The outcome of both generators is quite different. The Mono version uses public fields, the MS version use properties with back-up private fields. They both adorn the fields/properties with attributes from the XmlSerialization namespace.
For Microsoft it looks like this:

namespace BookStore.Books {
    using System.Xml.Serialization;
 
    ///
    [System.CodeDom.Compiler.GeneratedCodeAttribute("xsd", "2.0.50727.42")]
    [System.SerializableAttribute()]
    [System.Diagnostics.DebuggerStepThroughAttribute()]
    [System.ComponentModel.DesignerCategoryAttribute("code")]
    [System.Xml.Serialization.XmlRootAttribute("Book", Namespace="", IsNullable=false)]
    public partial class BookType {
 
        private string titleField;
        private System.Nullable priceField;
        private System.DateTime printedAtField;
        private System.DateTime soldAtField;
        private System.DateTime willReadTodayAtField;
 
        ///
        [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
        public string Title {
            get {
                return this.titleField;
            }
            set {
                this.titleField = value;
            }
        }
 
        ///
        [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified, IsNullable=true)]
        public System.Nullable Price {
            get {
                return this.priceField;
            }
            set {
                this.priceField = value;
            }
        }
 
        ///
        [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified, DataType="date")]
        public System.DateTime PrintedAt {
            get {
                return this.printedAtField;
            }
            set {
                this.printedAtField = value;
            }
        }
 
        ///
        [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified)]
        public System.DateTime SoldAt {
            get {
                return this.soldAtField;
            }
            set {
                this.soldAtField = value;
            }
        }
 
        ///
        [System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified, DataType="time")]
        public System.DateTime WillReadTodayAt {
            get {
                return this.willReadTodayAtField;
            }
            set {
                this.willReadTodayAtField = value;
            }
        }
    }
}

That could have been a bit more readable by removing all the fully-qualified namespaces (since there is a “using” at the top), the empty comments and the back-up fields (if you generate for .NET after version 3).

For Mono it is like this:

namespace BookStore.Books {
 
    ///
    [System.Xml.Serialization.XmlRootAttribute("Book", Namespace="", IsNullable=false)]
    public class BookType {
 
        ///
        public string Title;
 
        ///
        [System.Xml.Serialization.XmlElementAttribute(IsNullable=true)]
        public System.Single Price;
 
        ///
        [System.Xml.Serialization.XmlElementAttribute(DataType="date")]
        public System.DateTime PrintedAt;
 
        ///
        public System.DateTime SoldAt;
 
        ///
        [System.Xml.Serialization.XmlElementAttribute(DataType="time")]
        public System.DateTime WillReadTodayAt;
    }
}

Much more concise. But with an important omission: the Price has the right XmlAttribute, but should have been defined as “Nullable”. I’ve seen some remarks on this in the mailing-list/forum for Mono, but could not find out if this was already fixed. So I use the Microsoft XSD.exe whenever my XSD changes, copy paste the result to my MonoDevelop environment and recompile. I would rather call the MonoXSD on the background whenever my XSD changes (has anyone used scripts as Custom Tools in MonoDevelop?) so that the class gets re-generated automatically. Maybe the upcoming Mono 2.8 will fix it.

It’s all a DateTime

As you can see, all the fields are generated as DateTimes. Makes sense, since there is nothing else in the framework to hold time-related information. So what happens when you read an Xml file and serialize the Xml into a Book object?

Take this Xml:

<Book>
    <Title>How to be a Mac Developer</Title>
    <PrintedAt>1965-02-16</PrintedAt>
    <SoldAt>1965-02-16</SoldAt>
    <WillReadTodayAt>11:00:01</WillReadTodayAt>
</Book>

And process it with this code:

    class MainClass
    {
        public static void Main (string[] args)
        {
            string bookXml = @"
            <Book>
                <Title>How to be a Mac Developer</Title>
                <PrintedAt>1965-02-16</PrintedAt>
                <SoldAt>1965-02-16</SoldAt>
                <WillReadTodayAt>11:00:01</WillReadTodayAt>
            </Book>
            ";
 
            BookType book = (BookType) ToObject(bookXml, typeof(BookType), Encoding.Default);
            Console.WriteLine("PrintedAt:{0}", book.PrintedAt);
            Console.WriteLine("SoldAt:{0}", book.SoldAt);
            Console.WriteLine("WillReadAt:{0}", book.WillReadTodayAt);
            Console.ReadLine();
        }
 
        public static object ToObject(string xml, Type xmlObjectType, Encoding encoding)
        {
            MemoryStream xmlStream = new MemoryStream(encoding.GetBytes(xml));
            StreamReader reader = new StreamReader(xmlStream, encoding, false);
            XmlReaderSettings readerSettings = new XmlReaderSettings();
            readerSettings.CloseInput = true;
            XmlReader xmlReader = XmlReader.Create(reader, readerSettings);
            XmlSerializer xmlSerializer = new XmlSerializer(xmlObjectType);
            return xmlSerializer.Deserialize(xmlReader);
        }
    }

Different ideas on default dates

You get this output in Windows (running the exe build by MonoDevelop directly on my VMWare-WindozeXP):

And this output on the Mac:

Interesting difference, right? Windows sets the date part of “WillReadTodayAt” to DateTime.MinValue and Mono assumes it is today. It doesn’t matter that much, since I’m going to ignore the date-part anyway for that property.
The time part of the date-olny property “PrintedAt” is set to “12:00:00AM” in Windows and “00:00:00″ in Mono. Again, I’m going to ignore the time-part, but what will happen when I compare dates for equality? I prefer the Mono setting.

Invalid time data

Take this slightly modified Xml:

<Book>
    <Title>How to be a Mac Developer</Title>
    <PrintedAt>1965-02-16</PrintedAt>
    <WillReadTodayAt>2010-09-21T11:00:01</WillReadTodayAt>
    <SoldAt>1965-02-16</SoldAt>
</Book>

Running this on Mono result in an FormatException:
Fair enough, the data in the xml is not a Time, it is a DateTime. So an exception makes sense.

Wonder what Windows will do? After you have started an instance of Visual Studio to see the error, you get this:

So both platforms agree that the data is invalid. Unfortunately the position that Windows gives us is right after the offending data, at not before as I was expecting. So it took some time (way more than an hour) to realize that the problem was not in the “SoldAt”, but in the “WillReadTodayAt” just before it. Be warned.

Invalid date data

Now pass in a date-with-time on the field that expects only a date:

<Book>
    <Title>How to be a Mac Developer</Title>
    <PrintedAt>1965-02-16T08:01:03</PrintedAt>
    <WillReadTodayAt>11:00:01</WillReadTodayAt>
    <SoldAt>1965-02-16T09:00:02</SoldAt>
</Book>

Mono doesn’t like it. Same error as before, no indication on what line and what xml-tag, alas.
Windows doesn’t like it either. Same error as before, positioned right before “WillReadTodayAt”.

Conclusions

There’s a big difference in the C# classes that get generated. The Mono implementation is missing the support for nullables, the MS implementation is a bit too verbose.
The handling of incorrect date/time information is okay in both systems.
The date-part in a xsd:time gets defaulted to “1-1-1″ in MS and “today” in Mono. The time-part gets defaulted to “0:0:0″ in Mono and “12:0:0AM” in MS.

The code for this article

Storing data using plain ADO.NET

Ok, you might argue that there is no need to talk about plain ADO.NET, since nobody in his right mind is actually doing that anymore. Of course we -developers- have evolved and use document-databases like CouchDB and Raven DB or ORMs like NHibernate and Entity Framewok.

But every now and then you need code like the code below. In my case because we were building a demo and needed to be done quickly. Which we were not, because of the problems I ran into when trying to store null values in the database.

Book book = new Book {
	Id = Guid.NewGuid(),
	ISBN = "123-456-222",
	Title = "Through the looking glass"
};
 
using (SqlConnection connection = new SqlConnection ("some connections string")) {
	connection.Open ();
	using (SqlCommand cmd = connection.CreateCommand ()) {
		cmd.CommandText = @"INSERT INTO Book ([Id], [ISBN], [Title], [Publisher], [SuggestedPrice])
		       VALUES (@Id, @ISBN, @Title, @Publisher, @SuggestedPrice)";
 
		cmd.Parameters.AddWithValue ("@Id", book.Id);
		cmd.Parameters.AddWithValue ("@ISBN", book.ISBN);
		cmd.Parameters.AddWithValue ("@Title", book.Title);
		cmd.Parameters.AddWithValue ("@Publisher", book.Publisher);
		cmd.Parameters.AddWithValue ("@SuggestedPrice", book.RetailPrice);
 
		var rowsAffected = cmd.ExecuteNonQuery();
		if (rowsAffected == 0)
		{
			Console.WriteLine("No rows inserted!");
		}
	}
}

The sql to create the table is like this:

CREATE TABLE Book (
	Id			uniqueidentifier	NOT NULL,
	ISBN			varchar(255)		NOT NULL,
	Title			varchar	(128)	NOT NULL,
	Publisher		varchar	(128)	NULL,
	SuggestedPrice	money 			NULL
) ON [PRIMARY]

Finally the Book class is this:

class Book
{
	public Guid Id {
		get;
		set;
	}
 
	public string ISBN {
		get;
		set;
	}
 
	public string Title {
		get;
		set;
	}
 
	public string Publisher {
		get;
		set;
	}
 
	public decimal? RetailPrice {
		get;
		set;
	}
}
}

The reason to use SqlXXXX objects in stead of programming against interfaces like IDbConnection and IDbCommand is that I know I have a SQL Server database and will not change that in the course of this project and the fact that I have this nice simple syntax for setting parameters on the Command object:

cmd.Parameters.AddWithValue ("@Title", book.Title);

The AddWithValue-method is not on the IDbCommand interface, but is available as a public method on the SqlCommand class.

The book object comes from somewhere outside of this piece of code normally, and I cannot make any assumptions on the properties that are filled or not. For example the RetailPrice property is a nullable decimal, so it might therefore be null. No problem, since the database column SuggestedPrice is also nullable.

No problem? No problem in Mono. Run this code in MonoDevelop and store a book with a null value for the retaill-price and the publisher. Sweet.

A problem on Windows

Copy this code (or the compiled assemblies) to your Windows VM and run it there. You get an exception:

Why is that? Why would .NET complain about the Publisher parameter, suggesting it is not there? It contains a null value, right, but the parameter itself is there anyway.

So the message itself is confusing. It took me quite some time (and the use of the great SqlProfiler from AnjLab) to find out that the problem was in the null values. That must have happened to other people before. So Google to the rescue? Even that took some time before I found a thread on a PC Review forum from 2005 that suggested to go through all the parameters in your command to see if they are null or not and then set them to DBNull.

The platform independent version

Are you kidding me? Every time and everywhere I have a Command object and add Parameters to it, I must go through all the parameters like this?

foreach (SqlParameter parameter in cmd.Parameters)
{
    if (parameter.Value == null)
    {
        parameter.Value = DBNull.Value;
    }
}

And that has been like that since .NET 2.0? No improvements in the last 5 years? Probably, the reason you haven’t bumped into this problem lately, is that you have been using NHibernate (I did) and that solved it for you. I haven’t tried it with Entity Framework yet, but I guess that would solve it too.

I really think Mono does a better job here. When I set a parameter on a DbCommand object to null, I expect that object to translate that null value to something that the database understands. As is does for any other value.

Download the code

P.S. At one time in the process of finding a solution , I got so frustrated with Microsoft .NET and Visual Studio 2008 (since that somehow stopped building properly) that I decided to run Mono on Windows too. It took me 5 minutes to download and install Mono on my VM, then 5 more minutes to discover the existence of XPS (the Mono web-server) and I had my webservice running! Unfortunately, that was not an option for our production server, but to me it proved that the guys at Mono do a really good job.

I’m a long time Windows developer. Started with VB3 en FoxPro and such, and couldn’t imagine needing another plaform. Like the Mac that my geeky artistic brother-in-law loved so much. Windows was cool and I knew all about it and the Mac was for people that didn’t know how to use a PC.

The dawn of a new era

Then I came with my current employer and they provided a mobile phone and a laptop of course (like all companies in Dutch IT do). But this was about an iPhone and a MacBook Pro.
I decided to leave OSX on the machine, although I could have erased the disk and installed Windows Something, it’s got an Intel processor after all.
But leaving OSX on it, meant I had to install VMWare and run Windows and Visual Studio and such within that. And even though my machine is pretty fast, you notice the performance hit.

I kept trying, and started to like OSX and the applications on it. But what I liked most of all: hardly any updates and no reboots. I shut down my machine once a week or once every two weeks because I feel it’s something I should do. But I don’t have to: OSX keeps on running. Just close the lid, simply know that everything is suspended and know that everything will run as before when you open the lid the next day.

Amazing! I would shut down my Windows machine every night, just to make sure I had a stable system the next morning. My wife still has a Windows machine and to ensure that she makes backups every day, we configured the backup software to run “on system shutdown”. Because that is what you do every day, as a Windows user.

And then there was Mono

I’ve written about the Mono project before, albeit in the context of MonoTouch and iPhone development. Mono brings .NET to a lot of platforms, including Mac OSX. And it is really good. The Mono team is really close behind the Microsoft team in porting new API’s and Framework versions. .NET 4.0 is recently available on Windows, but Mono is already compatible.
And the amazing thing is that you can take your Windows assemblies and use them straight away on OSX! That might not be very imported for assemblies that you have the sources for, but it is very convenient for third party libraries you use, like log4net (although that has a Mono version) and Rhino Mocks.
And one of the best things the Mono team delivers is MonoDevelop: an excellent IDE for .NET development that will feel really comfortable when you come from Visual Studio.

Being a .NET developer, not necessarily a Windows developer

I still like the .NET framework, but I no longer feel that automatically implies I am a Windows developer. There are so many good tools that allow you to do everything on the Mac that you are normally doing on Windows. So I decided that in the new project I recently started (in which we will deliver on Windows) I will try to go as far as possible to develop my code on Mac OSX.
I will blog about my experiences in a series of posts. This first one is about connecting to SQL Server.

I can’t do without SQL Server

We will deliver our project an a SQL Server database, so it makes sense to use that during development also. It’s not that you have to, there are other options. If you have something like a nice Data Access Layer, you can run any SQLLite or MySql database and easily move to SQL Server later.
There also is a file-based no-install version of SQL Server these days that might run on Mono. Haven’t tried it yet, but could be promising.
But if you need SQL Server, you can still work in MonoDevelop in your comfortable Mac environment.

Inside the VM

SQL Server will run on Windows in your VM, but that will be the only thing you’ll need your VM for. SImply minimize the window after all the configuration is done and don’t think about it anymore.

Step one: opening up your Windows

You need access to your VM from your Mac, so you have to open up some things.

Go into Control Panel and choose Windows Firewall. Go to the advanced tab and add some ports. Port 1433 for SQL Server over TCP/IP and port 1434 for SQL Server over UDP. Maybe you can do without the UDP version, haven’t tried yet.

Step two: configure SQL Server

First, make sure the SQL browser is running.

Then enable TCP/IP in the Client Protocols

And check that the Default Port is on 1433.

In SQL Server Management Console open the properties of the server and make sure you have mixed authentication.

Step three: connect from your code or MonoDevelop

In your connection string use the IP-address of your VMWare instance. Something like

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<connectionStrings>
		<add name="dossiers" connectionString="Server=172.16.86.128;Database=mydefaultdatabase;User ID=sa;Password=bladiebla" providerName="System.Data.SqlClient"/>
	</connectionStrings>
</configuration>

That is! You can test your connection from MonoDevelop: Go to Tools/Database/Add database connection/SQL server:

Gebruiker

Ik heb de telefoon niet zelf bedient maar heb toch een redelijke indruk gekregen. Zelf ben ik een iPhone gebruiker en zie wel wat verbeteringen en interessante ontwikkelingen

• Het gebruik is opgezet rond hubs. Er is bijvoorbeeld een social hub, op deze hub komt alle sociale media bij elkaar en bieden ze mogelijkheden tot integratie. Er zijn dus geen aparte ingangen meer voor linked-in, contacts, facebook e.d. maar alle informatie wordt gebundeld weergegeven.
• Drie verplichte toetsen op de telefoon waarbij naast de enige toets op de iPhone om applicaties te starten er ook een zoek toets is die de standaard zoek functionaliteit heeft maar ook kan worden gebruikt voor zoeken binnen de applicatie. Daarnaast is er de back toets hiermee kan je terug naar de applicatie die een andere heeft aangeroepen. Een typisch voorbeeld waar ik mij vaak aan heb geërgerd is: Je leest je mail daar staat een link in die je opent en vervolgens zit je in safari. De enige manier om die te verlaten is stoppen en je mail opnieuw opstarten.
• Het metro concept van de user interface (Kort samengevat: beperkt grafisch en meer tekst) aangevuld met een panorama view en pivot view is wel een verfrissende aanpak waarbij je direct informatie ziet die je nodig hebt en niet alleen een menu.

Ontwikkelaar

Voor een .Net ontwikkelaar is er eigenlijk niets nieuws. Je kunt namelijk gewoon ontwikkelen met je opgedane kennis in XAML (WPF en Silverlight) in je vertrouwde ontwikkelomgeving. Ik wil echter wel een aantal punten benadrukken

• Een virtuele machine met de phone OS op de ontwikkel PC dus geen emulator of simulator dus betere test omgeving
• Naast API’s die telefoonfunctionaliteit geven zijn er ook API’s voor bijbehorende services in de cloud
• Er is een gratis omgeving bestaande uit visual studio express voor ontwikkelaars en Blend express voor designers

Het lijkt mij een eitje om de look en feel van de quote eetgids (een iphone app gemaakt door luminis) te bouwen voor W7

Business

• Beperkt aantal modellen en veel verplichte onderdelen zoals een grafische processor en sensoren als GPS e.d. Dit maakt de herkenbaarheid groter
• Geen exclusieve contracten met KPN’s en de TMobile’s
• Vergelijkbare appstore als Apple waarin een applicatie gegarandeerd binnen dagen wordt geplaatst.
• Office applicaties en zelf een sharepoint front-end

Het evenement

Het evenement zelf was wat mij betreft teleurstellend. Op een developer event verwacht ik concrete presentaties met duidelijke concepten en voorbeelden. Helaas bleven de presentaties erg oppervlakkig ook al aangemoedigd door de vele bizarre vragen die gesteld werden. Wat mij wel duidelijk is geworden dat Microsoft serieus hun best doet om weer aansluiting te vinden of zoals ze zelf beweren een voorsprong te nemen maar dat ze nog  heel goed hun best moeten doen om alles in de winkels te krijgen voor de kerst.

To build iPad apps, you need a couple of things that are all listed on the MonoTouch iPad page.

I couldn’t wait to build something that used the way bigger UI-surface that the iPad has. The first thing that attracted my attention when I fired up the Interface Builder was the UISplitViewController.
The idea behind the SplitView is that you have some navigation on the left (like a NavigationController, or just a TableViewController) and a data view on the right. That is, only when the display is in landscape mode. As soon as you turn the iPad (the iPad SImulator, of course) to portrait, the left side disappears and all the screen is available to the data view.
That behaviour is entirely taken care of by the UISplitViewController, at least if you stick to the conventions!

I decided to make a simple app with a list of places on the left side, and a map-view on the right:

The obvious start

When you start a new iPad application from the New Project menu, you get the well-known set-up of files in your project. Double-click the MainWindow.xib to fire up Interface Builder. Then drag the Split View Controller from the Library Window and drop it below the Window object in de MainWindow:

As you can see, you get a lot for free, including stuff you don’t want. Now it gets less obvious. After clicking Cmd-Backspace a thousand times and positioning my cursor everywhere, I had an epiphany. Since the SplitViewController won’t work without two other controllers I had to add something new before I could remove the old!

No kidding, that was really the solution. So I added a UITableViewController, Interface Builder magically removed the default NavigationController, and I added a MKMapView to the view-controller on the right.

This is the result:

I then added outlets to the AppDelegate for the map, the tableview and the splitview:

The less obvious code

To make the controllers react properly when the orientation of the UI changes, you need to override the ShouldAutorotateToInterfaceOrientation() method in each of your controllers. How do you do that?
The first step is to add a class to your project, make it inherit from your controller (a UITableViewController in my case) and override the ShouldAutorotateToInterfaceOrientation() method as below:

public class PlacesController : UITableViewController
{
	public PlacesController ()
	{
	}
 
	public override bool ShouldAutorotateToInterfaceOrientation (UIInterfaceOrientation toInterfaceOrientation)
	{
		return true;
	}
}

Repeat the step for the second controller:

public class Map : UIViewController
{
	public Map ()
	{
	}
 
	public override bool ShouldAutorotateToInterfaceOrientation (UIInterfaceOrientation toInterfaceOrientation)
	{
		return true;
	}
}

This should still compile. But it doesn’t work yet.

The even less obvious link from the code to the UI

The objects in the Interface Builder are standard objects. They need to be of the types that we just defined, to make the overridden methods work. See we need to make our own classes known to Interface Builder. That’s done by adding a Register-atribute and overriding the constructor with one that accepts an IntPtr.
The Map class changed in something like this:

[Register("Map")]
public class Map : UIViewController
{
	public Map ()
	{
	}
 
	public Map(IntPtr p) : base(p)
	{
	}
 
	public override bool ShouldAutorotateToInterfaceOrientation (UIInterfaceOrientation toInterfaceOrientation)
	{
		return true;
	}
}

The final step is setting the right class on the controller. Go to Interface Builder, select the UIViewController (e.g.) in the MainWindow, then go to the Inspector Window and type your classname over the default one:

And then it works! The map will re-orientate itself when the iPad is turned, and the Table View appears and disappears.
Of course you want some location data in the table and the map to show the locations when clicked on, but that’s all in the code you can download.

Enjoy making software for a device that makes you need to rethink the way you always made software…!!!

Mmmmh. re-reading that last line, I realize it’s a little hard to read. What I meant was something like Joe Hewitt wrote.

Op 30 maart organiseerde luminis samen met ArchitecIT een kennissessie over model-driven development. Aan de hand van vijf verschillende thema’s deelden sprekers van diverse organisaties hun praktijkervaringen met MDD. De volgende organisaties waren als deelnemer vertegenwoordigd: ArchitecIT, Delphino Consultancy, luminis, Ministerie van Defensie, Nedap, Nuon, PANalytical, Radboud Universiteit Nijmegen, Sogeti, Tennet en Thales.

De presentaties van deze avond zijn inmiddels online beschikbaar en kunnen hieronder worden gedownload.

In de praktijk zijn er bij softwareontwikkeling nog veel communicatie (overdrachtsmomenten) en bestaan de meeste ontwikkeltaken uit veel handwerk. Op basis van een MDD aanpak kunnen ontwikkeltaken worden geautomatiseerd en kan de onderlinge communicatie worden verbeterd. Hierbij is het echter wel belangrijk om te weten hoe MDD het beste kan worden toegepast en wat hierbij de meest voorkomende valkuilen zijn. Vanuit onze AgileMDD filosofie moet bij model-driven development een pragmatische en doelgerichte aanpak vooral centraal staan. Zo kan de bestaande ontwikkelkracht in de organisatie slimmer worden ingezet.

Programma kennissessie 30 maart:

• Agile MDD: huidige trends en ontwikkelingen – Deel 1 (Richard van der Laan)
• Agile MDD: huidige trends en ontwikkelingen – Deel 2 (Tony Sloos)
• Thales case: MDA in realtime heterogene systemen (Rene van Hees, Alexander Broekhuis)
• Defensie case: Microsoft DSL Toolkit in de praktijk (Gerrit Jan Timmermans, Erik Sanders)
• MDD en software architectuur (Angelo Hulshout)
• MDD kansen en risico’s (Andriy Levytskyy)

Wil je op de hoogte blijven van aankomende AgileMDD sessies of geïnteresseerd in advies op maat? Neem dan contact op met Inge Dokter (inge.dokter@luminis.nl) of bel 026-3653470.

Here we go.

What’s your problem, dude?

Well, we are all very Web 2.0 and so, using Ajax and jQuery and maybe even single-page-applications, but on almost any site I fill in a form, a textbox is just a plain textbox. You can type text, like digits and characters, even if they need only my age. Come-on guys, an age is expressed as a number, so why allow me to type in anything else but digits?
Sure, you validate at the server and maybe even at the client, but still I can make errors that I have to fix later. Please, help me by preventing the error up-front.

The solution, part 1: jQuery-plugin

The solution starts with a nice jQuery-plugin by Paulo P. Marinas called jQuery AlphaNumeric. There is also a nice MaskedInput-plugin, but I found that too limiting.
It is simple to hook it up to my textbox:

$("#mytextboxid").numeric();

And voila, nothing but digits allowed!

Cool, I want that on all the fields that I know are numbers, but I surely don’t want to repeat the above Javascript.

The solution, part 2: EditorFor

Starting with MVC 2 there is an Html-helper called Html.EditorFor(). It uses a lambda as a parameter and derives a lot of interesting information from that. If you type

        <%= Html.EditorFor(model => model.Length) %>

You get code generated like this:

<div class="editor-field">
<input id="Length" name="Length" type="text" value="7" /></div>

De ViewEngine determines that you have a Integer field, generates a textbox with the name/id of the field and sets the value. If you have client-side data-validation on, you get the extra span-tag for the validation message. More on that in a minute. Maybe this doesn’t look like much, but you get different output for different field types; like radio-buttons for a Boolean field and appropriate formatting for a Decimal field.
The EditorForModel() method even generates code for all the fields on your class.

The solution, part 3: Turning validation on

I was hoping that the EditorFor() method would be all I needed. It supports validation, both client-side and server-side, and (as argued before) what better client-side validation then preventing false input to begin with!
Alas, it doesn’t do that. But in combination with DataAnnotations (Scott Guthrie has a nice blog about that) it adds range-checks to your Integer-field. So if I can limit the characters that can be typed into my textbox with a line of jQueury and can check for the upper- and lower-limit of the value with client-side validation, then I’m happy!

But that means that I have to change the behavior of the EditorFor() method. Can that be done? Well……, sort of.

The solution, part 4: Convention over configuration

The nice thing about the MVC framework is that it takes convention over configuration, meaning you do not have to configure anything to get a working application, as long as you adhere to the rules/conventions. Nothing new for Ruby on Rails programmers maybe, but for me as a long-time Microsoft-ee it is close to revolutionary.

In this case, the interesting convention is that there can be a EditorTemplates sub-directory in my View directory. If it is in a normal View directory it works for only those Views, but if you put it in the Shared directory it works for all your views:

If you have a user-control in there named Decimal.ascx it will be used everywhere the EditorFor() encounters a Decimal property. There is a bunch of default user-controls that are described by Brad Wilson.

What I would have liked to be “The solution, part 5: Overloading the decimal template”

Since I like the server-side formatting of the default template, and only want to add some jQuery, I would have liked to do the following:

<%@ Control Language="C#" AutoEventWireup="true" Inherits="System.Web.Mvc.ViewUserControl" %>
<%= Html.EditorFor(Model) %>
<script type="text/javascript">
    $("#mytextboxid").numeric();
</script>

I simply delegate to the original implementation, passing in the data I have available. But that doesn’t work since the EditorFor() expects a lambda as a first parameter. And I don’t know any way to convert the data I have at that point in time to a lambda. If you know, please tell me and you’ll be my hero (hmmmm, I hope it’s not going to be that colleague coming up with the solution, that would really ruin my post…).

The actual “The solution, part 5: Overloading the decimal template”

The best I could do was copying the default template for the Decimal from the blogpost of Brad Wilson. Yes that’s a shame, copying code, having to maintain code that isn’t even mine, not profiting of the improvements Microsoft makes in the default template. Nevertheless, here it is:

<%@ Control Language="C#" AutoEventWireup="true" Inherits="System.Web.Mvc.ViewUserControl" %>
<script runat="server">
    private object ModelValue {
        get {
            if (ViewData.TemplateInfo.FormattedModelValue == ViewData.ModelMetadata.Model) {
                return String.Format(System.Globalization.CultureInfo.CurrentCulture,
                                     "{0:0.00}", ViewData.ModelMetadata.Model);
            }
 
            return ViewData.TemplateInfo.FormattedModelValue;
        }
    }
</script>
<%= Html.TextBox("", ModelValue, new { @class = "text-box single-line" }) %>
<script type="text/javascript">
    $('#<%= ViewData.ModelMetadata.PropertyName %>').numeric({ allow: '<%= System.Globalization.CultureInfo.CurrentCulture.NumberFormat.CurrencyDecimalSeparator %>'});
</script>

So, there is some server-side code that I copied, and then some client-side code I added myself.
There are two interesting things to mention about my own single line of code:

• The jQuery selector references the name of the property, thus guaranteeing that the javascript is coupled to the right text-box. Other solutions make you use a fixed class or prefix or postfix, but that only opens the possibility of name clashes. The name-property is available in the model-metadata. See the Bradd Wilson series mentioned above for more info
• The allowed character (the decimal separator) for entering money is retrieved from the CurrentCulture and not hard-coded. Of course.

As ScottGu would say: hope this helps.