Recently a customer requested a particle simulation with the intention to investigate correlations and the effect of multiple ‘attractors’. The real, underlying issue in this demand is the fact that certain situations and systems are governed by laws (differential equations) which cannot be solved exactly and require a dynamic system to search for critical coefficients. These coefficients result in (semi) stable configurations or highlight boundary conditions inside which things make sense.

The problem with these kind of systems is the scalability which has two aspects; the visualization which doesn’t scale well with the retained graphics pipeline of WPF/SL/WP7 on the one hand  and the O(N*N) calculation of interactions between particles.

Regarding the visuals there is really a lot to consider:

• if you inherit from a baseclass like FrameworkElement or Control or even UserControl you carry around with each and every particle a whole lot of stuff; data binding, styling, event and other infrastructure. While all these things are really great in an LOB, one doesn’t usually need all this when dealing with (lots of) particles
• typically the underlying surface needs to be a Canvas but this element also brings along many gears; the retained rendering system, the measure/layout mechanics, styling and more. Maybe the bonus you miss when bypassing the Canvas is the ability to easily integrate zoom and pan in an application.
• how much user-interaction is required, are mouse-events necessary or is it a ‘let it run’ system? Here as well, you need to consider carefully the implications (especially related to the scalability of the app).

With respect to the simulation or calculation part things are more straightforward but you also need to think about

• the possibility to use F# for the more mathematically oriented parts. The disadvantage is however the need to constraint your model to make sure things talk to C#, or you have to go the full nine yards in F# (talking to XAML is still a disaster in this case).
• how to optimize the calculations and to make sure that inside loops you break early and branch as much as possible
• dropping unnecessary properties of the particles and make the particle system as lightweight as possible

Within the Silverlight/WP context my experience is that if you use the Canvas/UIElement combination you easily hit a wall around 500 particles due to the way the rendering engine is functioning and the fact that the UIElement (or FrameworkElement) are too ‘heavy’. The inter-particle calculation will still run OK with 500 particles but the end-result is that it’s not a good solution altogether.

One can keep the Canvas as underlying surface and render particles as inherited Shape items but this doesn’t push the boundary much.

A step further is to replace the Canvas with a bitmap and use direct pixel printing. This is easily done with the WriteableBitmapEx project you can find on CodePlex. This boosts your rendering to 5000 particles if you drop interactions. However, if you switch on inter-particle forces you hit a calculational wall around 1500 particles which is still way better than the retained rendering solution but still somewhat underwhelming.

So, as a next step it would be great to use parallel processing for the algorithmic part but unfortunately there is no task parallel processing library (TPL) for Silverlight (though somewhat made an unofficial port). Another possibility would be to use MailboxProcessor of F# but it would entail threading issues and an altogether more difficult system.

Various samples and libraries are available for XNA and Windows Phone which allow one to create particle effects but

• the often use sprites or similar techniques to minimize the particle count. The results are usually very good but the aim in the context of games is usually to create a certain effect rather than to give a physically correct simulation.
• they often do not exhibit inter-particle forces. If they do the particle count is low.
• they focus on effects and not physics

See also the Flint-sharp library which is a port of the Flash/Flex library.

The conclusion is that one can perform simulations and develop particle system in Silverlight if the need is not above, say, a thousand particles. Above this threshold things become more difficult and Silverlight as a medium becomes less suitable for this type of task. Obviously there are high-tech solutions on the basis of C++ and GPU instructions but this was outside my client’s scope (and my technical know-how as well).

Note: on different level you need to note in the attached simulation sample that

• the average speed varies a lot while the amount of interacting particles stays the same
• the ergodic phenomenon if easily shown; that the average velocity over the sample equals the average of one item in time
• it’s quite difficult to create stable configurations; which mimics well how the real world is also constantly influenced by quantum fluctuations

References

• TPL library ported to Silverlight 4
• Particle system for games (XNA & WP)
• Flint for C#
• WriteableBitmapEx on CodePlex

Silverlight Integration Pack for Enterprise Library is a collection of guidance and reusable application blocks designed to assist Silverlight application developers with common LOB development challenges.

These blocks are designed to encapsulate recommended practices which facilitate consistency, ease of use, integration, and extensibility. The release also addresses the needs of those who would like to port their existing LOB applications, that already leverage Enterprise Library, to Silverlight.

Download Enterprise Library 5 for Silverlight

What’s in the box:

• Caching Application Block with support for:
  • In-memory cache
  • Isolated storage cache
  • Expiration and scavenging policies
  • Notification of cache purging
• Validation Application Block with support for:
  • Multi-level complex validation
  • Attribute-based specification of validation rules
  • Configuration-based specification of validation rules
  • Simple cross-field validation
  • Self-validation
  • Cross-tier validation (through WCF RIA Services integration)
  • Multiple rule-sets
  • Meta data type for updating entities with external classes in Silverlight
  • Rich set of built-in validators
• Logging Application Block, including:
  • Notification trace listener
  • Isolated storage trace listener
  • Remote service trace listener with support of batch logging
  • Implementation of a WCF Remote logging service that integrates with the desktop version of the Logging Application Block
  • Logging filters
  • Tracing
  • Logging settings runtime change API

  Exception Handling Application Block, including:

  • Simple configurable, policy-based mechanism for dealing with exceptions consistently
  • Wrap handler
  • Replace handler
  • Logging handler

• Unity Application Block – a dependency injection container
• Dependency injection container independence (Unity ships with the Enterprise Library, but can be replaced with a different container)
• Unity Interception mechanism, with support for:
  • Virtual method interception
  • Interface interception
• Policy Injection Application Block, including:
  • Validation handler
  • Exception Handling handler
  • Logging handler
• Flexible configuration options, including:
  • XAML-based configuration support
  • Asynchronous configuration loading
  • Interactive configuration console supporting profiles (desktop vs. Silverlight)
  • Translation tool for XAML config (needed to convert conventional XML configuration files):
    • Standalone command-line tool
    • Config console wizard
    • MS Build task
  • Programmatic configuration support via a fluent interface
• StockTrader V2 Reference Implementation (RI) (via a separate download)

How to get it

Both Enterprise Library 5.0 and Enterprise Library 5.0 Silverlight Integration Pack are to be available as Windows Installer Packages (MSI) via MSDN or as NuGet packages. The configuration tool is also available as a Visual Studio extension package (VSIX) from the Visual Studio Gallery. Today, you can NuGet and Visual Studio Gallery as the primary distribution channel, with the MSDN downloads following.

References

• Grigori Melnik’s blog on MSDN blogs
• The Enterprise Library on MSDN

• The usability aspect: how meaningful is a diagram with thousands of nodes, does the user still get insights with potentially a million links between so many nodes? Here are I have to think about how useless the Visual Studio 2010 architecture visualizer is if you try to display a big assembly. See the picture below which displays a small portion of the G2 architecture; 20.000 nodes and 90.000 links…useful? I don’t think so.
• The .Net framework aspect: out of the box the .Net classes related to WPF and Silverlight (the UIElement and FrameworkElement, that is) are ‘heavy’. These base classes contain stuff related to templating, databinding, triggers, animation and whatnot. It means that even if you don’t need any of this you have to carry around in the memory of your application a large amount of material. If you scale a diagramming solution to thousands of item it shows. If one really wants to display very large amount of data in a diagram there is no other way but to find alternatives to the retained graphics pipeline of .Net and go back to the good old low-level mechanisms.
• The granularity aspect: much like the usability aspect, is it meaningful to have a lot of animations and gimmicks when displaying large datasets?  I suppose if you wish to display large diagrams you are not interested in details and interactivity, or do you? Well, some customer do and want to have both a high-level overview and the ability to zoom down to a finer level of details. This is difficult to achieve with the current technology and average hardware. Being able to display a million nodes with databound entities and remain fluid is tough, the only solution being to switch in the zooming process from one algorithm to another. Much to say about this but it’s not the topic of this post.

In the end there are three solutions to these problems:

• Drop Silverlight/WPF as the technology in which you try to visualize something and move to C/C++ but in this way sacrificing a lot of magic  these frameworks offer. While sometimes a viable solution for the customer, usually not a solution I like to push too much for obvious reasons.
• Compromise between scalability and functionality by emphasizing the usability aspect; decision makers want clear diagrams and yet another cluttered interface is the last thing a company wants to pay for.
• Come up with an ingenious or new way of visualizing data. Often the type of data and the business context dictates here what ‘ingenious’ means, each type of data has its own peculiarities. So, finding a generic solution in this line is usually not easy.

While I describe all this in the context of diagramming, the remarks are valid for any data visualization solution. In the end, what matters the most to a customer is having a solution whether it’s a diagram or a different type of visualization often doesn’t matter.

Live Pivot falls into the last category and is a new data visualization approach which handles the issues mentioned above quite well even if the amount of data it handles is still moderate. No links or diagramming here but a way to traverse a multi-dimensional dataset by filtering out data through various widgets and pivot-like datagrams which are well-known from Microsoft Excel. The name ‘Live’ Pivot is hence aptly chosen as it displays pivot charts dynamically in function of the user’s criteria. I highly advice you to take a look and play with it, it’s a free Silverlight framework and I’m sure we’ll see interesting applications of it in the near future.

Now, the reason of this post is that although I was enjoying Live Pivot in the beginning I got more and more frustrated by the limitations imposed as time passed by:

• The datapool on which the visualization is based is a variation of XML called CXML. Though Microsoft created various tools to support the creation of CXML it remains a bit an awkward format if you want to work with data coming from WCF or use the visualization with your legacy data entities. Even if possible theoretically to convert your entities in memory using the supplied assemblies, it remains a bit of show-stopper when developing for Pivot.
• The visuals in Live Pivot are based on the SeaDragon technology which even for small amounts of entities will force you to have huge amounts of server space because for every image you need, say, a hundred copies of it in various resolutions. This is related to the Morton layout algorithm used internally to refine images as the user zooms on different levels of the dataset under consideration.
• Updating or serving collections from a database is possible but in practice no-one is doing this because it involves undocumented wizardry (see for example the cryptic explanation on the Pivot site). I’m sure things will become easier in the future though.
• The whole design and development process of Pivot collections is off the main development route, the framework doesn’t integrate well into a traditional N-tier application. In a way, creating a Pivot collection is more like the design of a Photoshop template for a website: you do it once and then try to not change it in fear of the consequences it has on the rest of the site.
• Live Pivot is freely available and well developed but if you wish to customize things for a customer…well, you can’t since the source is not yours.

Summary: Live Pivot is hardly a solution for the typical customer I meet. This said, I started to develop my own version of a Pivot-ish solution which would enable one to:

• Use WCF as a backend to serve collections and to handle large amounts of data (not limited by the client’s memory). One of our customer talks about terrabytes of records, how would one handle this in Live Pivot?
• Use standard POCO entities as data buckets. This goes hand in hand with the WCF requirement, of course.
• Use any XAML in the UI instead of only pictures. Let’s imagine you want buttons, tooltips, …in the display rather than just Morton-related images?
• Adapt the partitioning and visualization in function of customer needs.
• Have an iterative filtering of data outside the memory boundary. That is, use the whole SQL database backend as a datapool rather than just the entities in memory.
• Make use of all the fluid animation features of Silverlight
• Possibly combine diagramming and Pivot-ish visualization together.
• Run away from the horrible SeaDragon mechanics involving hundreds of little images
• The possibility to transpose the solution to WPF and have shared code WPF/Silverlight solutions

This Silverlight library is called ‘Twirl‘ and contains (as good as) all the features above. The things which still need to be worked on are the WCF integration and the dynamic querying of the backend.

While developing Twirl I discovered that a whole lot of features are really available out of the box in Silverlight (together with the SL toolkit, Expression Blend etc.) which made the development easy, in particular:

• The new fluid animation features of SL4 and Blend
• The ability to databind lists to any path, not just ItemsControl and ListBox
• The filtering and sorting of data through ICollectionView and  related classes
• The MVVM pattern with good support in Blend
• Easing functions, animations etc.

The biggest challenge was really how to partition data in several columns? That is; how do you organize an arbitrary set of data on the basis of a certain datatype into a limited amount of partitions? Here I had to go deep into unknown territories like data mining, clustering algorithms (k-means in particular), text analysis and more. At some point I developed an F# clustering library but had difficulties to map the F# vector type into C# types…and gave the interop effort up (it was a good learning exercise though). After much research and a lot of trials and errors I just developed my own view on the problem and a home-made algorithm came out of this. Currently the partitioning is quite good but more datasets and types of data need to be used in order to refine the algorithm.

Below are some screenshots of Twirl in various situations.

Twirl offers, much like Live Pivot but beyond it as well, the following opportunities:

• visualization of large datasets stored on SQL Server
• exploration of local drives through the out-of-browser or COM capability of Silverlight
• traversals of datasets which share a common set of properties
• interactive shopping carts and filtering of items
• cataloguing of ebooks or e-documents in general. Here we are looking into the integration with Windows indexing service and alike
• customization of the partitioning and the display
• integration with other visualization techniques (diagramming, charting and whatnot)

This Twirl development was inspired by a bigger customer project which also embraces Microsoft StreamInsight and correlation research of very large datasets. While the development of Twirl was mainly in function of our customers I realize it might benefit a lot from the input from others. Whether this means I need to put the code on CodePlex, I don’t know yet. Anyway, if you have interest in Twirl or want to share your thoughts, contact us.

The customer wants to see relationships between multiple data types, groups of data, drill down into it, have a geographical view, a temporal view and a conceptual view. And much more of course. That’s the moment you realize you spent ten years of your life creating all possible variations on the subject ‘diagramming’ and now you actually need a ‘diagramming database’. A whole different beast, not the usual yada-yada, what does it entail?

The thing is, relational database systems are not very suitable to act as repositories for graph-like structures but at the same time they are unavoidable because the whole world runs on them. Sure, there is something like an OO database but it would be even less suitable for graphs. So, you choose for a RDBMS even if you have some back-thoughts. At least RDBM systems are well supported on every API level and you get tons of fun tools which on the long run could be useful. In the case of SQL Server you know things like reporting services, StreamInsight, OLAP and whatnot will show up at the end of the road.

How do you store a graph-like system whose nodes are of an arbitrary data type? Say, you wish to store relationships between telephone numbers, file locations, car types, chat messages and credit card numbers. It wouldn’t be very wise to create data tables for each data type since you know at some point the customer will come along and ask for mobile numbers, addresses, book titles and cooking recipes! Well, there are not that many solutions in this case; you can use binary serialization of some data blob and store it as varbinary or you can use string-like serialization (which includes XML). Obviously, binary stuff in your database is not a good thing if you wish it to be searchable and ultimately open to other integrations. There is, hence, no other choice but to go for XML structs which at least is a well-supported standard across the while industry.

What about the relational structure? Easy enough, you just store the ‘from’ and the ‘to’ (sink, source, target, drain, start, end, initial, final…) in a two-field table. Voila, you’ve got a graph-database on top of an RDBMS.

Well, not really.Obviously having the data in place is just the beginning; you need to calculate a spanning tree, Hamiltonian cycles, Eulerian paths, shortest distance…and more fun pseudo-math stuff. If you want to use the standard algorithms it means you will put a lot of pressure on the data exchange with your client (say, a WCF service). Argh, you realize the only way is to actually do as much as possible on the database side. Through T-SQL or use .Net assemblies inside the SQL process? Not an easy choice. You start to wonder if creating a server is not a better option. But this means…writing a query processor, heavy threading programming and more wizardry which goes far beyond one’s standard toolbox. So…let’s assume T-SQL can do it. Actually T-SQL is not too bad if you are a bit fluent with it you can go quite far into this direction.
Status: a graph-database consists of two tables and a lot of smart stored procedures.

But hold on, the customer wants it also multi-user and multi-application. That is, the total data is not always ‘on’, only a subset of if in function of the application the current user is using. More branching, more subtleties in the stored procedures.
The situation is actually comparable to having heaps of XML documents where each user and each application adds attributes to the standard XML to decorate it with the necessary application-options and security settings. On top of that lots of tuples describe relationships between these XML documents. Did I mentions the relationships are also user and application dependent?

Maybe I’m giving you the impression that it’s an hopeless complexity and an overly open system which tries to embrace too much flexibility. It’s not that bad. In fact, we have a working prototype which includes a full-fledged client application (both for Silverlight and for WPF). The feeling sticks with me, however, that as far as abstract beauty is concerned a true graph database server would be better. This exists in Java but not in .Net and even if it would, I wonder how well adjacent systems (like reporting and such) would integrate with it. So, the current prototype is a balanced solution between various poles and constraints. In a next phase we’ll develop connectors to StreamInsight, file system, legacy event aggregators and more. The purpose being to be able to store and analyze graphs which cross the boundary of the local storage and also capture real-time events (through, for example, StreamInsight). Big fun and more challenging stuff is ahead.

A new Graphite sample (just click on the picture to see it live) has been added which demonstrates:

• Databinding of data coming a WCF RIA Service
• Expand/collapse of nodes from a given node with data from the RIA Service
• Labels on connections with data coming from the WCF RIA Services
• Customization of the nodes and databinding to the fetched WCF data
• Zooming and inflation (relaxation) of the diagram through databinding
• Custom controls inside the node to expand/collapse adjacent nodes
• General UI and Silverlight techniques (diverse utility classes are provided to help access resources, walk the visual tree and more)
• A custom container control to expand/collapse some info and a search panel
• Drag-drop from a control onto the surface (uses the DragdropManager of Telerik)

This was a hugely fun project to work on, the only minor detail was the deployment and I guess a lot depends on your provider and how easy it is for you to manage the settings of IIS. A few things I learned from this is:

• set the ‘copy to local’ to true to pretty much every assembly in your solution, don’t expect the assemblies to be present on the host
• set the security to anonymous on IIS and remove the “<authentication mode=”Forms”>” section from your web.config. Typically you’ll get a “The remote server returned an error: NotFound” message which doesn’t reflect the real issue. Also, trying to discover the underlying issues through Fiddler or Firebug will not reveal much.

Note: the data (company names, prices…) in the sample is random and is not connected to any real-world commodity-service. This sample demonstrates features of Graphite and is not meant to give any information on any real-world commodity.

You should also check out our latest mindmap implementation for Silverlight 4. Note that you won’t be able to see this application unless you installed Visual Studio 2010 beta since the Silverlight 4 plugin is not publily available (yet).

More innovation and Silverlight stuff coming up this year, stay tuned. We’ll also release the revamped website soon, hence the rather silent blog these days. You can, however, follow me on Twitter these days, it gives a more complete and colorful overview of what keeps me busy.

L2 G2 is our biggest product and has been used by companies to create ‘big’ applications in diverse markets. Because it is based on Unity for dependency injection and because Microsoft waited quite a while to release a Silverlight analog, it took a while to create a Silverlight analog of G2. We baptised it ‘L2′. Of course we could have used alternative injection libraries (many have popped up this year in fact) but we wanted to stay as parallel as possible to G2 in order to enable our customers to move with ease between G2 and L2. So, it’s there and you can test it online here. It’s still in beta and we welcome any feedback. You can also download the code and the core library, see the L2 product page for more.

S#scripting library. This little jewel allows you in effect to alter an application at runtime much like macro’s. In fact, you can compare it to VBA in office except that our S# library will work in any .Net context (including XNA, Silverlight, WPF and WinForms). I will talk about S# more later this week in another blog posting and Denis Vuyka will also publish soon more information regarding S# and, another kid on the stage; Processing#.

iSee! In addition to this we have been experimenting with novel approaches to diagramming and how to take full advantge of Silverlight and WPF animation capabilities. This lab project which has been now already for one year in the back of our minds is ready to be tested as well. Head over to our iSee demo and let us know what you think. iSee is all about animating visual elements and you can apply it anything which inherits from the FrameworkElement base class, that is…as good as anything you’ll find in WPF and Silverlight. Furthermore, iSee is really easy to use. It boast a fluent API and a very intuitive series of methods. The fluent things means you can chain actions ad infinitum.

For example, the following line of code:

Attach(image, true).Scale(3, 3, Duration.Automatic)
.Move(new Point(145,150)).Scale(0.3, 0.3, Duration.Automatic).Play();

will attach a given image to a canvas, scale it with a factor of three, subsequently move the image to another location and un-scale it to the original proportions. Easy! It would take a substantial amount of work to code this in C# or to create an equivalen storyboard in Expression Blend.

We also wanted to enable certain diagramming scenario’s and have integrated diverse methods which allow you to create diagrams, mindmaps and data visualization with just a few methods. In the demo you will see a few examples of this.

SPGLast but not least, I’m also proud to announce our new Silverlight property grid. Much like L2 is an analog of G2, so is the Silverlight property grid (we call it ‘SPG’) an analof of our WPF property grid (‘WPG’). You can read about it on Denis’ blog and watch this space for more in the next few weeks.

All these products represent a considerable amount of work (testing, demo versions, docs, samples…) and we are still in the process of packaging them. Also, for prices and official releases you’ll have to wait a bit, we’re working on it. Meanwhile let us know what you think. The Silverlight market is booming these days and there are excellent products around, still we think that our set of products fill a gap and represent a unique spectrum of solutions you’ll not find elsewhere.

It’s been a while since I jotted something down on this blog…it’s been hectic for months now and the pace doesn’t seem to slow down but I think I have a few interesting things to mention.

Since the end of last year G2 has grown bigger, stronger, more popular and various big customization projects are on our workbenches. At the same time, Graphite continues to be our bestseller but most customers use it ‘as is’ without much alterations. There are however a few lines of evolution. On the one hand, Silverlight becomes more and more popular and seems to have a better market penetration than WPF. This means that also Silverlight diagramming solutions are increasingly asked for. In this context, Graphite allows an easy path to data visualization but misses the flexibility and breadth of a true diagramming framework. Equally well (and enforced by the Silverlight wave), more and more customers wish to have a dual solution; all the advantages of diagraming in WPF and the possibility to bring as much as possible on the web (Sharepoint integration, read-only viewers and so on). These evolutions and the obvious gap in our products related to this meant a necessary investment in Silverlight and a common diagramming core. Now, hold your horses and don’t expect the sky since Silverlight still remains a subset of the .Net framework and even though Silverlight v3 broadens the horizon I don’t think any day soon you’ll have a full-compatibility tale (except through XBAP, but that’s another story). In any case, here is the good-news show and some notes from the field where diagramming meets Silverlight, the Entity Framework, MySQL and WCF.

Before describing the juicy technical details let me give you an overview of the application (screenshot above) we’ve been working on recently. It’s a Silverlight application which allows one to visualize any kind of relations between different types of entities; persons, documents, sites, etc. Any kind of relation means that it allows multiple links and generic graphs but can also handle tree-like data and multiple diagram layouts. The database uses just a few tables to store the graph structure and the actual data (persons, documents, metadata) can reside in the database but could also be stored in another database, in XML files, you name it. The application also allows one to edit relations and to edit the entities; you can draw links and it will be saved in the database, you can change or add entities and so on. Finally (and this is what this blog items is mostly about) this Silverlight application can also be used as a WinForm application and accesses data through a local self-hosted WCF service. The WCF service is a RESTful ADO.Net data service which can also handle standard HTTP requests and thus acts as a local webserver. Minor features of this application is the integrated security system which can be connected to the ASP.Net membership mechanism, to ActiveDirectory services or any legacy single-sign on system you have in-house. Let me emphasize that this application boasts a full-fledged diagramming framework for Silverlight. which in many ways extends/surpasses our current Graphite product. For example, just for fun I developed in less than a day a reasonable mindmapping application for Silverlight (screenshot below).

Entity Framework

First off, I have been happily surprised by the Entity Framework (EF). I know, it’s got a long list of shortcomings and I’ve walked the whole avenue of POCO adapters, detached entity tricks, constrained Visual Studio designer and more. However, in the context of Silverlight I was pleased to experience the ease with which it worked on top of MySQL and through the ADO.Net data services (aka Astoria). Initially I wrote the whole framework for SQL Server and had later on to move to MySQL…oh miraculus! Only minor changes were necessary on the database level (no Guid’s in MySQL, no triggers e.g.) and one needs at least MySQL v5 to make it work. Note that this happy message has more its origins in the hard work which went into developing the .Net driver for MySQL rather than how open Microsoft is towards other relational database systems. In fact, I can also claim the same for other systems: EF works well with SQLite and with SQL Server Compact v3.5. There are pro’s and con’s with each and you need to tweak your model because of datatype differences, missing features (views, stored procedures etc.) but I expected a complete no-go orginally.

ADO.Net Data Services

Concerning ADO.Net data services. It works flawless with the EF, whatever the underlying database-dependent model is. In fact, if the data type signature is identical in different database systems, you only need to change one word in your code to switch from one to the other:

1
2
3
4
5
6

[System.ServiceModel.ServiceBehavior(IncludeExceptionDetailInFaults = true)]
public class CrystalMsSqlService : DataService<crystalmysqlcontext>
{
...
}
</crystalmysqlcontext>

This code is the generator of a REST WCF service on top of a EF model. The CrystalMySQLContext is the data context generated by the EF. If you wish to switch to, let’s say, SQL Server you only need to replace the data context specification and you’ll be safe for the rest of the journey.
Note, as a side note, the ServiceBehavior attribute which propagates exceptions to the surface and which is not added by default in Visual Studio. You will also want to add a try-catch in the code to convert the exceptions to a DataServiceException like this:

1
2
3
4
5
6
7
8
9
10
11
12

public static void InitializeService(IDataServiceConfiguration config)
{
    try
    {
        ..
        config.SetEntitySetAccessRule("SomeTable", EntitySetRights.All);
    }
    catch (Exception e)
    {
        throw new DataServiceException(e.Message);
    }
}

It will make debugging much easier, trust me.

Why ADO.Net data service (ADS)? You need to know that Silverlight only likes asynchronous data exchange with the outside world. Now, in order to expose data via WCF you can go via a artisanal, hand-crafted WCF service but ADS makes things easy and turns an EF into a RESTful service in a snap. In effect, it means you have all your tables and views available in Silverlight as if they were on the client. The price for this is the asynchronous behavior, which at times can be frustrating; you cannot have a data request and a data related action in the same C# method. A wisdom which I can give to you here is: don’t try to launch inner join requests from Silverlight to the REST, it doesn’t work. If you wish to do this you need to create a view in you database and make this view accessible in ADS as if it was a table. This simple fact makes SQL Server Compact immediately a bad choice because it doesn’t support views (SQL Express will do however).

WCF (ah, so much beauty!)

One of the things which kept me busy for a long time was how to demonstrate/ship a Silverlight demo without having to ship a database and without having to buy an expensive SQL Server license? Imagine you have a beautiful Silverlight application for a customer but you don’t want a whole setup involving a SQL Server database deployment (logins, connection strings etc.) and you also don’t want a complicated IIS setup wherein the Silverlight, WCF, XAP…reside. What to do? At first I tried to develop a local WCF proxy which would go to a file based data system. This meant a self-hosted WCF service, some sort of database on the client and a separate local webserver. This worked well up to the point that Silverlight kept on complaining about security issues. No mystery here: Silverlight doesn’t want to access data sources if they are not coming from the precise (includes the port number!) same origin as the xap (i.e. the Silverlight application).

Easy enough, I thought, but read on. You only need to supply the client with the clientaccesspolicy.xml or the server with the crossdomain.xml. Extend the WCF service like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

[ServiceContract]
public interface IWebServer
{
    [OperationContract, WebGet(UriTemplate = "/clientaccesspolicy.xml")]
    Stream GetSilverlightPolicy();
    [OperationContract, WebGet(UriTemplate = "/crossdomain.xml")]
    Stream GetFlashPolicy();
...}
public class CrystalMsSqlService : DataService<crystalmysqlcontext>, IWebServer
{...
public Stream GetSilverlightPolicy()
{
    const string result = @"< ?xml version=""1.0"" encoding=""utf-8""?>
                                <access -policy>
                                    <cross -domain-access>
                                        <policy>
                                            <allow -from http-request-headers=""*"">
                                                <domain uri=""*""/>
                                            </allow>
                                            <grant -to>
                                                <resource path=""/"" include-subpaths=""true""/>
                                            </grant>
                                        </policy>
                                    </cross>
                                </access>";
    return StringToStream(result);
}
 
public Stream GetFlashPolicy()
{
    const string result = @"< ?xml version=""1.0""?>
                                < !DOCTYPE cross-domain-policy SYSTEM ""http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd"">
                                <cross -domain-policy>
                                    <allow -access-from domain=""*"" />
                                </cross>";
    return StringToStream(result);
}
...
}
</crystalmysqlcontext>

It doesn’t work. Or at least, you can check that the WCF service does supply the policies but for some reasons Silverlight in my case still didn’t want to query the WCF service.

Now, because REST allows a client to query a WCF as if it’s a website and because one can serve both ascii and binary data through WCF/REST the obvious next step was to extend the WCF service with methods which would mimic a webserver behavior. Easy to do in fact by extending the service with more methods and templating the URL requests:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

 
[ServiceContract]
public interface IWebServer
{
[OperationContract, WebGet(UriTemplate = "/clientaccesspolicy.xml")]
Stream GetSilverlightPolicy();
[OperationContract, WebGet(UriTemplate = "/crossdomain.xml")]
Stream GetFlashPolicy();
[OperationContract, WebGet(UriTemplate = "/app.htm")]
Stream GetApplicationPage();
 
[OperationContract, WebGet(UriTemplate = "/{pagename}")]
Stream GetSitePage(string pagename);
[OperationContract, WebGet(UriTemplate = "/Images/{imagename}")]
Stream GetImage(string imagename);
[OperationContract, WebGet(UriTemplate = "/Crystalizer.One.xap")]
Stream GetCrytal();
 
}
public class CrystalMsSqlService : DataService<crystalmysqlcontext>, IWebServer
{...
public Stream GetSitePage(string pagename)
{
    try
    {
        WebOperationContext.Current.OutgoingResponse.ContentType = "text/html";
        string fileContents;
        string path;
        path = string.Format(@"WebRoot\{0}", pagename);
 
        using (StreamReader sr = new StreamReader(path))
        {
            fileContents = sr.ReadToEnd();
        }
        return new MemoryStream(Encoding.UTF8.GetBytes(fileContents));
    }
    catch (Exception)
    {
        return null;
    }
}
 
public Stream GetImage(string imagename)
{
    byte[] fileContents;
    fileContents = File.ReadAllBytes(string.Format(@"WebRoot\Images\{0}", imagename));
 
    // byte[] resp = Encoding.ASCII.GetBytes(string.Format(HttpHeader, "image/png", fileContents.Length, fileContents));
    return new MemoryStream(fileContents);
}
...}
</crystalmysqlcontext>

This now turns your RESTful ADS service into a webserver which serves both the data, the site pages and site images.
One last trick is the fact that because now your WCF service actually implements two interfaces the default endpoint configuration of ADS will not work. To solve this in the self-hosted scenario you can add another endpoint as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

private static void HostWithWebServiceHost(Uri baseAddress)
{
 
    WCFHost = new WebServiceHost(typeof(CrystalMsSqlService), baseAddress);
    WebHttpBinding binding = new WebHttpBinding();
    WCFHost.AddServiceEndpoint(typeof(System.Data.Services.IRequestHandler), binding, "dataservice");
 
 
    WCFHost.AddServiceEndpoint(typeof(IWebServer), binding, "").Behaviors.Add(new WebHttpBehavior());
    ServiceMetadataBehavior smb = new ServiceMetadataBehavior {HttpGetEnabled = true};
    WCFHost.Description.Behaviors.Add(smb);
 
    WCFHost.Open();
}

At this point the Silverlight application will run at the client without any server or internet connection.. You can go a step further and even host the Silverlight application in a Windows form which at the same time hosts the WCF service, but that’s really a no-brainer. You simply add a webbrowser control in a WinForm and call the HTML page which contains the Silverlight control and you’re done. Note that by doing this you have a Silverlight 2 application which runs as if it’s a regular WinForm application. Still, because the WCF service serves ANY client you can also access the Silverlight application through a standard browser. Actually, this fact helps to debug the Silverlight code by attaching the debugger to the browser process. I have not found any other way to debug Silverlight if it’s not part of a Visual Studio web application (as in my case).

In my final ‘show it to the customer without deployment’ solution, I removed the database from the client setup and kept the MySQL on a publicly available location. The only things you need to watch in this case is the fact that by default EF will store the database connection credentials in the web.config. You need to create a EF compiled connection with the EntityConnectionStringBuilder and override the CreateDataSource method of the ADS service:

1
2
3
4
5
6
7
8
9
10
11
12

protected override CrystalMySQLContext CreateDataSource()
{
    EntityConnectionStringBuilder entityBuilder = new EntityConnectionStringBuilder();
    entityBuilder.Metadata = @"res://*/MySql.CrystalMySQLModel.csdl|res://*/MySql.CrystalMySQLModel.ssdl|res://*/MySql.CrystalMySQLModel.msl";
    entityBuilder.Provider = "MySql.Data.MySqlClient";
    entityBuilder.ProviderConnectionString = "server=xxx;user id=xxx;password='xxx';persist security info=True;database=illumineo";
 
    EntityConnection con = new EntityConnection(entityBuilder.ToString());
 
    CrystalMySQLContext ctx = new CrystalMySQLContext(con);
    return ctx;
}

MySQL

During the testing and deployment phase of this project I encountered various difficulties which were all related to the fact that all the assemblies you assume are not necessarily on a blank machine. Adding these assemblies to the bin directory doesn’t solve these issues, the great trick here is a configuration you need to add in order to tell the CLR these assemblies are available. During the installation of a database driver the machine.config is altered but this is not the case if you deploy a self-hosted service and the binaries are not in the GAC:

1
2
3
4
5
6
7
8
9

< ?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <dbproviderfactories>
      <clear />
      <add name="SqlClient Data Provider" invariant="System.Data.SqlClient" description=".Net Framework Data Provider for SqlServer" type="System.Data.SqlClient.SqlClientFactory, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
      <add name="MySQL Data Provider" invariant="MySql.Data.MySqlClient" description=".Net Framework Data Provider for MySQL" type="MySql.Data.MySqlClient.MySqlClientFactory, MySql.Data, Version=6.0.2.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d" />
    </dbproviderfactories>
 
</configuration>

Silverlight only likes it async

Concerning the asynchronous behavior of Silverlight I can only say: you need a different mindset to live with it. In my Silverlight application I invested quite some time to create an MVC pattern on the basis of asynchronous data access. Because Silverlight handles templating/styling differently and doesn’t have a databinding mechanism which support element names all of this was tricky at times but in the end I think I was able to cook a very clean and felxible Silverlight framework which in fact has not much affinity with diagramming and could be used as a generic framework (just like our O2 framework for WPF).

Commercial rands

I mentioned in the introduction that one of the aims of this project was to have a common core between WPF and Silverlight. Bad news here: you cannot share assemblies between Silverlight and WPF, you cannot share template, you cannot share databinding patterns, you cannot…well, lots of things. You can find on the net different ways to handle these difficulties (pragma’s in your code for instance) but I haven’t found any of these satisfactory. So, the end result in this context is a big deception, until Microsoft brings on stage a clear strategy targetting this scenario you have to live with duplicated code. For our diagramming aims this means we have a completely different Silverlight library which covers a lot of the features available in G2 but is nowhere near identical in code as G2. On the surface (read: our commercial offering) we now have

• an enterprise level diagramming framework (product: G2) for WPF and a MEF-based framework for WPF applications (product: O2).
• a very complete diagramming framework (product: L2) for Silverlight and a MVC-based framework for Silverlight applications

which in look-and-feel will give you pretty much the same experience. A BPM modeller or a workflow application in Silverlight with the same functionality in WPF…it’s all possible but not on the basis of a common core. In commercial terms this means that a customer wishing to have a dual Silverlight/WPF product needs to pay twice for customizations but can share a common data access layer hidden behind a WCF facade. Speaking about commercial moves I should also mention that The Orbifold has acquired recently two new products which were previously in the open source realm: Denis’ property grid for Silverlight and WPF, as well a Petro’s S# scripting framework (aka Script.Net). More about this in the near future.

There is so much more to say about our other projects and our latest WPF magic but I’ll stop here and leave if for another blog post.

It took me little effort to shift my WPF experience to Silverlight and to transpose some layout algorithms. There are some difference, for sure, but hardly anything I can complain about. Alltogether I truly enjoyed the hours I spent on developing this little ‘applet’. Graphite is very similar to the Java applet but the implementation is more clean and I have taken much care of the API. The only significant part is the layout algorithm, which you can almost copy/paste from the Java applet or fetch from some Netron code. Hyperbolic layout and tree layout are easily implemented but I’m not sure I will, because I have to focus my attention back to GraphSquare.

See for yourself:

• the black version
• the green version

Usage:

• use CTRL-click on nodes or on the canvas to add random nodes
• use SHIFT-click to select nodes before creating a new edge. Note that you can select only two of them.
• use ALT-click to delete a node and its bindings
• hover over a node to see the underlying info

Features:

• things are pretty much all encapsulated into a custom Canvas object
• the API and events make it easy to manipulte the diagram in any context
• the architecture is model-view based, the model being a straightforward graph data structure
• data can be imported from flat text files or from XML
• the visualization is based on WPF/SIlverlight templating and styling features outside the control, which means one can customize colors and appearance. The flexibility of WPF’s triggers and other subtleties is missing in Silverlight but one can get quite far nevertheless.
• the layout algorithm is continous, which means you can drag nodes around like beans in jam. It all behaves very ‘organic’.

Addendum: due to a different font on your side you might see node titles a little bit bigger or smaller than on the screenshots.
Demo: go ahead and have fun with the Graphite demo.