SQL Server Triple Store

by billkrat 13. May 2012 08:43

While working with the IRing ISO15926 team (charter group member) I learned a great deal about the Semantic Web, TripleStores, Triples and the SPARQL query language.  Since we were using the Joseki server (Java) there was not much I could take away to use with new contracts/clients as they were/are on the Microsoft Windows platforms using SQL Server; an environment where there are no tools or query languages for triple stores which is probably why it has not been embraced by SQL developers. 

I am starting a new adventure in my efforts to create such tools which will be available on my http://SolrContrib.CodePlex.com project.   I am developing an application for my wife’s business which will have “big data” requirements.  She generally takes 1000-1500 pictures and hours of video footage per event and her events go back over 9 years; I want to provide her clients online services while preparing her for growth of her business which will one year include a building to book events in (we’ll need terabyte servers).  The goal is to have a data platform that utilizes a mix of SQL Server and Solr (allowing each to do what it does best, i.e. Solr for faceted searches and SQL for FileStream storage of video/pictures).   The key will be to devise implementation for an interface which will allow the same SQL Syntax to be used by both platforms (transparent to us Microsoft SQL developers); based on requirements we can choose SQL, Solr and/or both from our data layer.

I am in the early stages of development and within an Agile spirit (get core functionality done first) I am going to have a slight departure from http://www.w3.org/TR/rdf-primer/ by staying away from urls (harder to read at a glance) and I’m going to create my own taxonomy so that I don’t have to study/learn existing ones.   Later this won’t require any code refactoring; at most a conversion utility that will update my data to reflect available industry standards.

Below is the prototype data I am starting with to develop the core stored procedures; I’m shooting for full functionality using SQL first, so that I can generate Unit Test, and will later create implementation for Solr utilizing the same interface – when all my unit test pass for the Solr implementation then configuration settings can determine what platform will be used for each data layer.

Note: the very cool thing about TripleStores is the ease in which data can grow.  For example I added a “Domain” to the App entity (Predicate field) so that I can store Website information for the applications.   I only had to update the Application.usp_TransposeList stored procedure (source below) and add my new data. 

001-Data

Triples are a new way of thinking and it will take some time to wrap your head around it.  It has been years since I worked with them so I’m taking a progressive approach, taking baby steps, so that I can logically retrieve stored data without getting lost.

If a picture says a thousand words I can let the output from my initial stored procedures do a lot of the talking.   I will note that with the emergence of new technology (Windows 8) I’ve had to scrap all libraries and have effectively started over.  We currently only have two applications, a memorial site for my father and a desktop application for my wife’s business (which actually will span into Web and Phone applications reusing desktop code).

I figured I start with something simple by looking for ComboBox content such as supported application types (Web and Desktop) as well as Applications (LoveBob and PhotoVideoClientMgr).  I then went into requesting a list of users that are registered to a particular application (Bill and Diane), next I went after meta data attached to a specific user, and finally I obtain a list of all email addresses for the user (myself) – all from the data above stored as triples.

note: all of the SQL Code to generate the following follows the image
002-QueryWindow

Not all processes will be generic, there will have to be some hardcoding, i.e., I have to transpose the Application information into a table for easier processing.   If the process is an enterprise level process, such as “Application”, then I’ll add a schema for it and put all specific processing in it.  Below is the

ALTER PROCEDURE [Application].[usp_TransposeList]
    -- Add the parameters for the stored procedure here
    @Subject nvarchar(MAX),
    @Predicate nvarchar(MAX) = 'App'
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

; with rootCte as (   
    select    Predicate +'/'+ Object as rootKey
      from  Triples.TripleStore
     where    Subject = @subject
       and    Predicate = @predicate
), mainData as (
    select    Id, Subject, Predicate, Object
      from  Triples.TripleStore
     where    Subject in (select rootkey from rootCte)
)
select  -- Transpose the triple store into table for application
        MAX(Case when Predicate='Name' THEN Id Else NULL END ) as [Id],
        MAX(Case when Predicate='Name' THEN Subject Else NULL END ) as [Subject],
        MAX(Case when Predicate='Name' THEN Object Else NULL END ) as [Name],
        MAX(Case when Predicate='Type' THEN Object Else NULL END ) as [Type],
        MAX(Case when Predicate='Domain' THEN Object Else NULL END ) as [Domain]
from mainData
group by subject
END

The Triples schema will hold all of the functionality for processing a TripleStore, I use a Synonym for the table so that I can later (easily) swap it out for the production table.

ALTER PROCEDURE [Triples].[usp_RootLevel]
    -- Add the parameters for the stored procedure here
    @Subject nvarchar(MAX)
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

    select    Id, Subject, Predicate, Object, Predicate +'/'+ Object as rootKey
      from  Triples.TripleStore
     where    Subject = @subject
      
END

ALTER PROCEDURE [Triples].[usp_FirstLevel]
    -- Add the parameters for the stored procedure here
    @Subject nvarchar(MAX),
    @Predicate nvarchar(MAX)
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

; with rootCte as (   
    select    Predicate +'/'+ Object as rootKey
      from  Triples.TripleStore
     where    Subject = @subject
       and    Predicate = @predicate
)
    select    Id, Subject, Predicate, Object
      from  Triples.TripleStore
     where    Subject in (select rootkey from rootCte)
END

ALTER PROCEDURE [Triples].[usp_SecondLevel]
    -- Add the parameters for the stored procedure here
    @Subject nvarchar(MAX),
    @Predicate nvarchar(MAX),
    @Type nvarchar(MAX)
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

; with rootCte as (   
    select    Predicate +'/'+ Object as rootKey
      from  Triples.TripleStore
     where    Subject = @subject
       and    Predicate = @predicate
)
    select    Id, Subject, Predicate, Object
      from  Triples.TripleStore
     where    Subject in (select rootkey from rootCte)
       and    Predicate = @Type
END

The adventure begins, the saga continues….

Tags:

Triples

Windows Kinect–HRESULT: 0x80070016 error

by billkrat 21. April 2012 09:12

Environment
Visual Studio 2011 Beta
Windows 8
Samsung Slate

I started development on my Gwn.Kinect.Desktop library with the Depth stream.  I was surprised to find that approximately 20% of the time when I started my application the Kinect would generate a HRESULT: 0x80070016 error, red light would display for a few seconds, and then the sensor would shut down – I had to disconnect and reconnect to get it working again.  This happened with no consistency other than it would consistently happen, i.e., sometimes it work work 6/7 times in a row and others it would fail after the first or second launch of the application.

It is important to note that this is being developed independent of a UI, I am developing for maintainability, extensibility and reusability.   As the framework nears completion I will plug in the UI components so I have no idea if the initialization techniques that I was using (from the book identified below) will even work. I should note that the fix was to apply initialization techniques used in the SDK examples. 2012.14.12 Update I have the ColorStream plugged in (WebCam) to verify components are working – change set 97421.

2012.14.12 UPDATE – Although this blog demonstrates how to avoid slow initialization times I have been unsuccessful eliminating the HRESULT issue running in Debug mode.  I have to emphasize “Debug mode” because I had been running the Kinect Explorer (demo) from runtime executable with no issues (so I was assuming I was doing something wrong).  Turns out that after loading the Kinect Explorer source, and running in debug mode (F5) I could not run the Kinect Explorer more than 5 consecutive times without an HRESULT error.  I did find that I could consistently “start without debugging (Ctrl+F5)” without the error so I tried this with my SolrContribDesktop application and so far have not been able to get it to crash (I tried 10 consecutive times – this is a record).   So far my workaround will be to run without debugging (Ctrl-F5) and then attach the debugger. Crashes when starting without debug (Ctrl+F5) also…

Microsoft Bug Reported HERE <== UPDATED 5/14/2012, Microsoft reported that this is a Windows bug.

HRESULT
Figure 1 – HRESULT error information

In running the sample applications that accompanied the Windows Kinect sensor I noted two things – the sensor initialized considerably faster and it consistently loaded 2012.04.22.. It was time to debug this issue….

My http://SolrContrib.CodePlex.com project (change set 97373) has a SolrContribDesktop application using the (decoupled) Gwn.Kinect.Desktop module.  This is a Multi Targeted application, however the Kinect module will only be available to desktop apps (since Kinect hardware is not supported by Metro).

The KinectPresenter of this module (using the MVP-VM pattern) is currently configured to implement the Depth and Skeleton features of the Kinect (not ColorStream).   Below I show the code for the command that will handle initializing and starting of the Kinect sensor under this configuration:

 SensorCommandR1
Figure 2 – Command that configures and starts sensor

Since my code followed the patterns indicated by the book **Beginning Kinect Programming with the Microsoft Kinect SDK (pg 10 for ColorStream and pg 50 for DepthStream) with no success, in regards to the initialization steps, I decided to add the missing ingredients that seemingly allowed the Kinect Explorer sample to work (the Kinect Explorer accompanies the SDK).  I enabled the color stream and used constructor parameters to see what would happen.  I was pleasantly surprised to find that not only did my crashing occur less frequently, now the sensor was initializing with the same performance that I was seeing in the Kinect Explorer demo!!!   To appreciate the difference you should watch the video clip by clicking on figure 4 and THEN watch the video clip shown in figure 5, you will see the performance I’ve been dealing with since I started development of the Kinect – it is significant difference!!!

** I strongly recommend purchasing this book if you are going to do any serious Kinect programming – I found it invaluable in the learning/developing process.  Visual Studio 2011 Beta was released since this book was written and many things have changed (been improved).  Note: I found that their downloaded samples have the constructor parameters.

Note: The highlighted changes in the upper panel in figure 3 are the ONLY changes I made to get the performance and reliability benefits, they are executed “before” the KinectSensor.Start() is called on line 308.
 SensorSourceR1
Figure 3 – Sensor.cs source code excerpt

Note: The sensor starts initializing when the IR Projector (red light) turns on.  You will find that during the initialization process it will turn on, turn off and then turn back on – it is then ready for use.   After enabling the color stream the delay between the times the projector turns on is significantly shorter.  See the difference between video clips by clicking figure 4 and figure 5.

Figure 4 - After enabling ColorStream and using constructor parameters

After watching the following video in figure 5, watch the video in figure 4 again to appreciate how much faster it is.

Figure 5 - Without the ColorStream enabled (no constructor parameters)
Figure 6 - Kinect will shut down and report “Not Powered” after crashing (20% of the time)

Tags:

Kinect

Kinect SkeletalViewer: Cannot open include file: 'SDKDDKVer.h': No such file or directory

by Administrator 14. April 2012 04:19

The solution is 3 easy steps...

While evaluating C++ development for the Kinect (for my http://SolrContrib.CodePlex.com project) I ran into issues compiling the newly installed project from the Sample (on a fresh install of Windows 8 and Visual Studio 2011).   In the end I found that "the juice wasn't worth the squeeze"; the C# demos performed just as well as the C++ demos (in the same task that will meet my concerns) with a fraction of the coding (and learning curve).  

Binging a solution wasn't productive - I am a C# developer (with no understanding of the C++ environment) trying to compile a project under Windows 8 using Visual Studio 2011.  I learned early that when you live on the bleeding edge that sometimes you have to bleed, I was hemorrhaging.

I did however find an excellent clue within the VS11: C++ include directories not set correctly in new project bug reported on the Microsoft site.  There was enough information to let me know that I required a "Windows Kits" folder (WindowsSdkDir) which will hold the SDKDDKVer.h.   Long story short the Bing trail took me to the following:

 

Windows Driver Kit (WDK) 8 Consumer Preview: http://msdn.microsoft.com/en-us/windows/hardware/hh852362 

  • Step #1 - Download and install this development kit, I wasn't trying to create Windows Drivers, however I felt it was safe to assume it would have everything Windows Drivers developers required to compile their C++ applications.   The assumption paid off; after I installed it, along with the Windows Driver Frameworks (WDF) Co-Installers it will recommend that you install, the folders were now available!

 

But I was still not compiling!  I went back to bug report and like Jennifer I did not have the required "/shared" folder in the Include directories path (which she manually added).  HOWEVER, I did note a checkbox setting [ ] inherit from parent or project defaults as I stumbled around trying to find "HOW" includes worked in this environment (using the following steps):

Step #2 - comply with the following steps

  1. Right click on SkeletalViewer Project
  2. Select Properties
  3. Open Configuration Properties (ref SkeletalViewer Property Pages in image below)
  4. Selected VC++ Directories (not shown in image)
  5. Select Include Directories (shown in image below)
  6. Click the dropdown box (down arrow icon) that will appear to right of input field
  7. Select Edit (you will see Include Directories windows shown in image below)
  8. Click the Macros button and scroll down to the WindowsSDK_Include path (shown below)

I saw how the $(WindowsSDK_includePath), included with "Inherited values", contained my /Shared folder!!!

I clicked the Checkbox Inherit from parent or project defaults, exited all windows applying updates as applicable. 

I Hit compile and  I got another error - this time for a .lib file (can't recall what it was). 

Step #3 - I followed the same steps from 5 on for the Library Directories path.

I Hit compile and Walla!  I can compile and run the KinectViewer application! 

Tags:

Kinect

Portable Class Libraries under Metro WinRT

by Administrator 7. April 2012 18:39

Source code: http://SolrContrib.CodePlex.com change set: 97145

The beauty of Composite Applications is that they are resilient to change; in reading Scott Hanselman's blog, and having commented on my experiences with Portable Class Library (PCL) issues with Metro, I was informed by Scott, David Kean and others that the issues were resolved with the beta release.  Considering the sources, I refactored my http://SolrContrib.CodePlex.com application and, with David Kean's help, I was able to get my PCL port of Unity (and new port of Prism) plugged into the framework in a short period of time.  

In the figure below, the Portable Class Libraries are highlighted in green. I pulled out the MetroIOC container (the only IOC container available for the Metro developer preview) along with the Prism bits and replaced them with the following three portable class libraries: Gwn.Library.Prism.Portable, Gwn.Library.Unity.Portable and Gwn.Library.Portable.   I was very impressed with the power of the PCL framework - the required reflection components were available to create ports of both Prism and Unity (scaled down significantly which is to be expected).  The power of the PCL framework is SIGNIFICANT because it permitted me to refactor most of the framework API classes, which were dependent on Prism and IOC, into the Gwn.Library.Portable classes - the only classes that I couldn't migrate were those that had dependencies on Windows components such as Controls and Visibility; they remain in the Gwn.Library.Desktop and Gwn.Library.Metro projects as applicable.

I love the fact that PCLs will work on all platforms.  I went through some rough cycles with Multi Targeted development; I first started with developing to the least common denominator but it turns out that these are the most fragile, i.e., on new releases Silverlight or Metro projects (the primary source code) won't load but the desktop applications always seem to survive the process.   This time I place the source code in Desktop applications and have the Metro (and upcoming Phone) projects link to them.   I will develop to the most common environment (desktop) and deal with compile issues during compile time.  Since I'll have Metro projects linked to the desktop source, issues won't get past me I just won't have the benefit of intellisense to tell me that I'm coding in a manner that will not compile on a different platform. 

As for Silverlight... I remember going to a FoxPro conference in which Microsoft dominated the conference with the topic of this thing called .NET.  I then got to watch Visual Basic get pushed off the cliff  - deep down I had to smile because as a FoxPro, objected oriented programming, developer I was always feeling like the step-child (to VB), now I see .NET moving in the direction of FoxPro (with Linq queries against data), which I jokingly refer to as FoxPro 11, anywho's I'm digressing...  I see the warning signs and since my target audience will be Windows 8 and its IE browser won't support plug-ins (Silverlight) it is a moot point to travel that path.  Instead I'm going to dig deep into JavaScript projects and ASP.NET MVC4 when it comes time to stand up the web components of my framework - components that will reuse all existing architecture.

 

Building Enterprise Composite Applications in C# for Metro WinRt

by BillKrat 3. April 2012 17:36

Composite Applications 

 

Source code, documentation and UML diagrams: http://SolrContrib.CodePlex.com

See Video clip of application discussed in this blog HERE

Prism developers wanting to build composite Metro style applications for the enterprise environment will be faced with some challenges; it could be some time before the Patterns and Practices (P&P) teams can update the projects we rely on, i.e., Prism and Unity.  In the absence of such tools we’ll need to roll our own extensible, maintainable and scalable framework comparable to what we were accustomed to.   Fortunately, the P&P teams provided guidance that has proven to withstand the test of time (even in a highly agile environment); looking to meet the challenge I rolled up my sleeves and used their Microsoft Application Architecture Guide (MAAG) (http://AppArchGuide.CodePlex.com) to build a framework that complied with most, if not all, of the principles it contains.   The resulting framework (that this article is based on) is available in the open source project http://SolrContrib.CodePlex.com (change set 96548).  The Gwn.DataBinding.Demo application (a proof of concept) resides within the solution, it looks and feels very much like Prism and uses a “Unity like” Inversion of Control (IOC) container that is based on Ian Randall’s open source project at http://metroioc.codeplex.com/; without his IOC container this undertaking would not have been feasible.  Note: I added the IUnityContainer interface to his container, pulled in the Unity unit test and performed test driven development (TDD) until I achieved basic functionality.  Once completed, I was able to pull in the Prism classes that supported event aggregation and commands so that I could perform TDD using Prisms unit test.

The MAAG architecture guide, published in 2009, is best described by the reputable Microsoft personnel involved in its creation:

The purpose of the guide is to help solution architects and developers to design and build applications on the Microsoft platform that are more effective, to support key decision making at the early stages of a new project, as well as providing topic-specific content to help architects and developers improve their existing solutions. This guidance incorporates the contributions and reviews from more than 25 external experts and customers.  - S. Somasegar, Senior Vice President of Developer Division

The guide is based on a number of key architecture and design principles that provide structure. It includes guidelines for identifying and dealing with key engineering decisions, and an explanation of the quality attributes, crosscutting concerns, and capabilities that shape your application architecture; such as performance, security, scalability, manageability, deployment, communication, and more. - Scott Guthrie, Corporate Vice President of .NET Developer Platform

The field of application architecture and design is dynamic and constantly evolving. The foundations on which successful solutions have been built in the past will continue to serve us well into the foreseeable future, but we should also expect that the pace of innovation, in both technologies and new design approaches, will not decrease. The Microsoft platform and the .NET Framework and the range of technologies and scenarios that they support are both deep and wide, and getting deeper and wider all the time. On the other hand, we don’t need to wait for what might be. We can build compelling valuable solutions right now, and hopefully this guide will help you do just that.  - David Hill, patterns and practices, September 2009

Since data binding is the basis for communications between the WPF View and ViewModel it seemed an appropriate (necessary) place to start with my new enterprise composite framework.  I downloaded the Windows 8 Developer Preview DataBinding sample project (not available in the Beta) and used it to create the business rules that would be used by my first Metro application.  I would take the seven scenarios and apply the “Key Design Principles” as outlined in Chapter 2 of the MAAG.  It is beyond the scope of this article to cover all of the design principles covered by the MAAG, but I will cover the basics of the more applicable principles.  Note:  you will find that I quote the MAAG throughout this article.

I started with “Minimize upfront design”, a seemingly under-rated topic with agile development but something I firmly believe in.  I was pleased to find the UML architectural tools readily available within Visual Studio 2011 developer preview.  Within the guidelines set I came up with the following design (figure 1) which clarified the shared and overlapping responsibilities for the seven scenarios in the Microsoft DataBinding sample project.  The use case diagram adequately covered separation of concerns, single responsibility principle, principle of least knowledge, and don’t repeat yourself (DRY) principles discussed on page 11 of the MAAG.   The diagram also served as a means to conduct a gap analysis which will provide an understanding of what the existing application does while outlining the components required for the new design.  

clip_image001

Figure 1 Sample overview

Within the spirit of “The Composite Approach”, as outlined by chapter 1 of the Prism 4.0 – November 2010 guidance, that states the following:

An effective remedy for these challenges is to partition the application into a number of discrete, loosely coupled, semi-independent components that can be easily integrated together into an application “shell” to form a coherent solution.  Applications designed and built this way are often known as component application

My shell (MainPage.xaml in Figure 2) will have a MenuRegion, MainRegion and StatusBarRegion.   The MenuView and StatusBarView will always be visible with only the active scenario being visible at any given time within the MainRegion.   The framework will be responsible for loading the seven modules, MenuModule, and StatusBar module’s views into their respective regions while providing a decoupled means of communication between components (using IOC event aggregation); more specifically communication between layers, each scenarios presenter with the status bar presenter, and Menu events.  

clip_image002

Figure 2 MainPage.xml

Where both the Model-View-Presenter (MVP) and Presentation Model (MVVM) patterns are discussed in the MAAG I opted for the MVP approach as it is one I learned early in my Prism design (reference my MSDN article on MVPVM at MSDN for more details).  In Figure 1, I breakdown the MVPVM triad for the Menu, Status bar, and Scenario 6 (in Figure 1) showing their relationship with the Business Logic Layer (BLL) and Data Access Layer (DAL) so that I can “be explicit about how layers communicate with each other”.

To comply with the guideline “Do not overload the functionality of a component” while applying the single responsibility and separation of concerns it encompasses, it was important for me to breakdown the responsibilities of the various ViewModels as shown in figure 3.   All of the scenarios have a reset button thus all ViewModels will derive from ResetViewModel.   Since scenarios 1 and 2 don’t have properties to bind to they will have instances of ResetViewModel.  Likewise scenarios 4 and 5 do not have properties outside of those available on ScenarioViewModel so each will have an instance of it.   Scenarios 3, 6 and 7 have unique requirements so each will have to have their own ViewModels.

The application of the above noted principle allows us to easily reuse available components making it the responsibility of the presenters to populate the ViewModels or handle reset button commands as applicable.

clip_image003

Figure 3  ViewModel overview within the context of their MVPVM triads

With the basic high-level architecture out of the way I could then get to the business of data binding.   Although the original DataBinding sample project didn’t provide a “last reset” time stamp, I figured I’d add this feature and placed the property on my ResetViewModel as shown in figure 4 line 26.   Note that unlike the DataBinding sample I don’t subscribe to the “Click” event of the button (from code-behind) instead I use the Command and CommandParameter (lines 11 and 12 in top pane of image 4).   The ResetViewModel will derive from the Gwn.Library’s MvpVmViewModel in an effort to “use abstraction to implement loose coupling between layers”.   The MvpVmViewModel wires-up the command (line 50 of figure 5) which will raise a ProcessEvent (line 105 of figure 5).   Each ViewModel simply notifies the application it was clicked – it doesn’t concern itself with subscribers of the event allowing it to remain loosely coupled.  

clip_image004

Figure 4 Reusable ResetButton control and its respective ResetViewModel ViewModel

clip_image005

Figure 5 MvpVmViewModel base class; command, wire-up and handling (publish)

Presenters that are interested in the Reset button click event need only override the GenericCommandLocalHandler method (a function of a shared ScenarioPresenter base class) and handle it as applicable.  For example in figure 6 we show that Scenario 1 will use Passive MVP to directly reset the three view sliders to default values.   We all are aware that we could use data binding (slider to ViewModel properties) however the exercise of this application is to honor the intent and business rules of the original DataBinding sample which directly sets the view controls – this was another reason MVPVM was a good choice for this solution because this is not possible with Presentation Model as the ViewModel will not have a reference to the View.  

I could have used a more traditional approach by creating a ResetCommand, versus reusing a GenericCommand property on the ViewModel, and in the presenter’s initialize method bind my button to a HandleResetCommand method in the presenter.  Instead I opted to “define a clear contract for components” as they are reusable and I want to “establish a coding style and naming convention for development” that will “lead to better maintainability”, i.e. Without looking at my code you now know exactly where every reset button click is handled within each scenarios presenter; in the respective Presenter’s GenericCommandLocalHandler.    Note: naming conventions dictate that a LocalHandler will only handle its modules button clicks versus overriding the GlobalHandler which handle “any” reset button click (any module).

clip_image006

Figure 6 Reset button handler

Now that we have a very high level understanding of the application and are armed with the knowledge that our Presenters will populate our ViewModels with data, wire-up event handling for event aggregation, and handle the reset button clicks in the applicable presenter’s GenericCommandLocalHandler (allowing our view’s to handle user interaction via the MVP Supervising Controller pattern), we need to discuss how the framework is bootstrapped (started up).   A process that was recently changed with the Visual Studio 2011 Beta released, requiring me to perform a little refactoring to get the Gwn.Databinding.Demo application working again.  

The traditional Prism approach to bootstrapping an application is shown in Figure 7 (pointed to by the arrow); we assign an instance of the view (implements IShell) directly to the Windows.Current.Content which gives us a hook into the process as we can instantiate the bootstrapper and allow it to load/initialize modules and return us a completely hydrated view (populated with its hierarchy of applicable views) before we assign it to Windows.Current.Content and activate it.

clip_image007

Figure 7 Bootstrapper process using Prism

In contrast, the new process used by Windows 8 has us instantiate a new Frame (reference line 50 in figure 8) and assign it to the Windows.Current.Content.  Since line 51 expects a type we lost our hook that permitted us to provide an instance of the MainPage view; by line 54 the view will have been instantiated and is available within the Frame instance.  Fortunately this challenge was not new to me because this is similar to how the Windows Phone 7 works.  So this was a challenge I was forced to solve when I ported Prism and Unity to Windows Phone (Silverlight), my MSDN article on how to “Build Multi-Targeted Apps for the Desktop, Prism and Windows Phone 7” goes into this in more detail.  My solution is shown in the sequence diagram in Figure 8.  Unlike the rest of the framework, where the Presenters will be responsible for instantiating the views, I will have the main view (also known as the Shell) constructor instantiate the bootstrapper and run it, which in turn will populate the Shell’s regions (figure 2) with the applicable hierarchy of the applications views; a bootstrapping process I will cover in more detail below.

clip_image008

Figure 8 Bootstrapper sequence diagram

With the reusable components available it is now time to plug them into the enterprise solution.  Our framework will use Prism concepts, this requires a Bootstrapper class to define the modules and register implementation for applicable interfaces - an excerpt from the Bootstrapper.cs is shown in figure 9.  The BootstrapperBase class will iterate through the configured modules, execute the modules Initialize method and once completed will execute each modules ModulesInitialized methods.   To understand the following code (which is essentially Prism and Unity) it is important to understand how Inversion of Control (IOC) works.   Note: I have a blog on the topic of IOC at http://www.global-webnet.com/Blog/post/2011/10/13/Unity-Dependency-Injection-Primer.aspx if you require more information:

clip_image012

clip_image010

The configured modules in figure 9 start to reveal the power of enterprise composite application development.  The Gwn.DataBinding.Demo application, which consists of nothing but reusable components, simply has to register available modules within the enterprise for the application.   If Scenario1Module was a User Management module and I required it in another application I would simply add it to that enterprise applications bootstrapper.   Perhaps the new application has different business rules for the existing User Management module since it will access the new Cloud data source – but only for the new application (so you can’t affect existing applications that use the module).  You would simply have to register the applicable (different) business logic layers and data access layers as applicable, i.e. instead of EmployeeBll and EmployeeDal referenced in Figure 9 you might register the EmployeeCloudBll and EmployeCloudDal classes for the implementation of IEmployeeDll and IEmployeeBll respectively; as long as you implement the interfaces as required for the unit test to pass you can swap these out without touching the reusable modules which means no regression testing of existing applications since they have not been affected – no code was changed for them; only the new BLL, DAL and application have to be tested.

clip_image014

clip_image016

To further clarify the framework bootstrapper process we will cover how each of the modules configured in Figure 9 interact with the main shell so that our components identified in figures 1 and 3 can communicate with each other and function in a decoupled manner (each module can stand on its own and be reused within other applications).

Before continuing this would be an appropriate place to discuss “separation of concerns”.   In a real world enterprise application I would not combine my Infrastructure, layers and modules within the same project. 

For this demo the Infrastructure folder referenced in figure 10 holds the contents of what would normally reside in the Gwn.DataBinding.Infrastructure project; it contains components that are specific to this application which cannot be easily reused in other enterprise applications.  

The layers folder, which contains the BLL and DAL components for the enterprise, would reside in either one or many projects depending on the enterprise, i.e. large enterprises may want to separate them into individual projects so that the smaller applications don’t have to pull in the “kitchen sink” to simply access a user module.  The same holds true for the Modules folder and the modules that it contains.

Each of the module folders referenced in figure 10 contain a module component, this class is what the bootstrapper framework instantiates.   It is the modules responsibility to instantiate the presenter(s) it will use and identify the module as a menu option if applicable.  In figure 11 line 25 we show how “setter injection” can be used to instantiate the Scenario1Presenter (top pane of image) and “constructor injection” is used to instantiate the Scenario2Presenter (lower pane).  You’ll find that the MainModule (bottom of figure 10) uses setter injection to instantiate both the Menu and Status bar presenters.

clip_image018

Figure 11 Setter and constructor injection of Presenters

It is then the Presenters responsibility to instantiate the view and ViewModel that it is addressing the concerns for.   In Figure 12 we could have used the Presentation Model (MVVM) approach and placed the ModulesInitialized code (on line 36) within the ViewModel but then this would have tightly coupled our ViewModel to the concerns of this particular menu process which is to retrieve the menu options from the ModuleCatalog thus forcing us to violate the “separation of concerns” and “do not overload the functionality of a component” principals defined in the MAAM.   Instead we can reuse an existing MenuView (shown in Figure 12) and our implementation for IShellViewModel  which in this case was configured in our bootstrapper (figure 9) to use the ShellViewModel, neither of which have any knowledge of the ModuleCatalog or each other.  Our MenuPresenter (figure 12) may be the only presenter within any enterprise application that chooses to use the ModuleCatalog for these shared resources.   Note how the ModuleCatalog on line 23 of figure 12 has no dependency on any implementation of IModuleCatalog; we are using the frameworks default implementation but could easily replace it at the framework, application or module levels by simply registering our new implementation – our presenter will use the default (or new implementation) as applicable.   This feature of IOC composite applications allows us to have full control over our individual modules (using shared components) without impacting or creating dependencies on those shared components.

clip_image019

Figure 12 Menu presenter and view

Scalability and maintainability of enterprise solutions are not the only benefits of composite applications.  You also have high velocity extensibility even in the face of inexperienced (new) developers in an environment that is notorious of high learning curves (Prism and IOC).   If a new developer was tasked to create “Scenario #5” for our demo application with the following business rules:

· Use data results from team list using shared Team BLL/DAL

· Provide reset capabilities that will deselect list box results (provide date and time stamp)

· Notify status bar when user selects a team

They could do so in minimal time, without understanding any of the complexities of Prism (composite framework) or IOC; they simply need to know how to create a new module (the Template module holds barebones MVPVM components), how to register and instantiate their implementation of interfaces (in bootstrapper), and understand the framework.

clip_image020

Figure 13 Scenario #5

With little to no code in the Module and ViewModel of the MVPVM components their Scenario5Presenter could look as follows:

clip_image021

Figure 14 Scenario 5 Presenter

Since the Reset functionality is core functionality consistent throughout the entire application the ScenarioPresenter<View, Presenter> base class (line 19 of figure 14) will handle that logic for us for free, we only need to perform the following steps (reference figure 14):

· Line 26 - resolve our BLL using constructor injection (configured in the bootstrapper in figure 9)

· Override our Presenters GetViewModelData() method, with the knowledge it will be called at the appropriate time, requesting our data asynchronously using the TeamBll.List() method.  We’ll take this opportunity to update our ViewModels LastReset value.

· Override the DataArrivedBllHandler() and assign the results to our ViewModels TeamList property (line 49) while updating our date and time stamp.

· Override the ViewModelPropertyChangedHandler (line 61), prepare message (line 67) and publish the message to the status bar (line 73).

The application framework can be learned without assistance (if well documented) or easily taught by more experienced developers allowing the new developer to be instantly productive (increased velocity) while affording them the opportunity to take on the heavier learning curve at a time that has no impact on critical timelines.

In conclusion, composite Metro applications, as demonstrated by the demo application (using Prism and Unity “like” plumbing), that support the architectural MAAM principles discussed in this article will have the following characteristics:

· Scalable

· Extensible

· Maintainable

· Decoupled

· Reusable components

· Modules with a clear separation of concerns / single responsibility

Which all contribute to increased velocity, stability and the success of existing and new Metro enterprise applications that will be supported by all Windows 8.

 

How to Compile Solr source code - required tools and steps

by BillKrat 1. April 2012 05:20

I'm in the process of porting SolrJ to C# for my http://SolrContrib.CodePlex.com project - time to dive into the Java world!

Compiling Solr is not a complicated task, however it does require experience that us Microsoft developers may not have.  The result is that what should have taken a couple of hours took a few days to get figured out.  I suspect it would have taken longer if I hadn't learned a great deal from the ISO15926 team (I was member of the charter team for my Silverlight experience); I learned how to use SVN and got my feet wet in the Java world (Joseki server and triple stores).  

Within the context of making it easier for the next guy, I took all of the lessons learned on my development laptop (Windows 7) and decided to see if everything would work on my development HP Touchsmart w/Windows 8 - it did!  I used the opportunity to document the downloads and steps. 

Word Document version of this blogs content: SolrSetup.docx (904.43 kb)

  1. Development IDE: http://www.eclipse.org/downloads/ - Java EE developers
    • Copy eclipse folder to c:\Java
    • Executed Eclipse:
      • Workspace prompt, input => c:\java\EclipseWorkspace
  2. I already had the Java JRE installed (may have impacted steps, i.e., 6c)
  3. Java Development Kit (JDK) http://www.oracle.com/technetwork/java/javase/downloads/index.html
  4. TortoiseSVN - http://tortoisesvn.net/downloads.html
    • Executed TortoiseSVN installation msi
      • Selected “command line client tools”
      • Created c:\java\LuceneSolr folder
      • Right click c:\java\LuceneSolr – selected “SVN Checkout”
      • Solr website for source code: http://lucene.apache.org/solr/versioncontrol.html
        1. Input (fig 2): http://svn.apache.org/repos/asf/lucene/dev/  Clicked button
        2. Selected lucene3661
        3. Clicked “Head” button (then clicked OK to accept default)
        4. Clicked OK (exited Head wizard)
        5. Clicked OK (exited Checkout window)  download starts
  5. Linux environment: http://cygwin.com/ - setup.exe
    • Root directory c:\Java (root when within Cygwin)
    • Package directory c:\Java\Cygwin (installation files)
    • Add “curl” and click on All icon (see arrow in figure 1), required to import xml data.
  6. Ant - http://ant.apache.org/bindownload.cgi
    • Unzip apache-ant-<ver>-bin.zip to c:\java\Ant
    • Add bin folder to path (fig 3)
    • Note: Ant will not execute at this point!  You must copy the tool.jar file from its location in the JDK to the JRE<ver>\Lib folder (fig 4)
    • Test Ant from command prompt – should get “Build.xml does not exist!” message.
  7. Compile application
    • Navigate to c:\Java\LuceneSolr folder
    • Type “ant compile”  (fig 5) – there will be a number of warnings (patient may be required the first time)
    • Navigate to the c:\Java\LuceneSolr\Solr folder
    • Type “ant dist” - you must do this from the Solr source code folder.  Ant will create a “dist” folder and place the compiled .war and .jar files in it.
    • Navigate to the c:\Java\LuceneSolr\solr\dist folder and copy the apace-solr-<ver>-SNAPSHOT.war file to the c:\Java\LuceneSolr\solr\example\webapps folder (fig 6) – be sure to rename it to solr.war
  8. Start application
    • Navigate to c:\java\LuceneSolr\solr\example
    • Type “java –jar start.jar” (fig 7)
  9. Import Data
    • Launch CygWin application using Icon generated on desktop (fig 8)
    • Type “cd /” to change to root folder – you will find the contents of the Java folder if you followed my folder structure naming conventions.   I find this layout makes navigation easier as you pretty much follow the same steps that you would from a command prompt.
    • Type “ls-al” to view contents of folder (optional)
    • Type “cd LuceneSolr/Solr/example/exampledocs”, note - if you do an “ls-al” here you will find all of the sample .xml files.
    • Type “./post.sh *.xml” to import all sample data into the currently running solr core.  Note: if you did not install “curl” in step 5c this will not work.   The “.” before the post.sh command is important, without it the environment will not find the file (unlike MS command prompt the current directory is not considered in the default path).
  10. Launch Solr

 


Figure 1 - Cygwin package selection - "curl"


Figure 2 - Source code repository

 


Figure 3 - Add Ant bin to path


Figure 4 - Copy tools.jar to Jre7/Lib

 


Figure 5 - Compile the application

 


Figure 6 - Copy compiled war file to webapps folder

 


Figure 7 - Start Solr using Jetty

 


Figure 8 - Launching CygWin


Figure 9 - using post.sh to import files

 

SolrSetup.docx (904.43 kb)

Tags: ,

Solr

Proof is in the pudding - adding a button, wire-up, execute, and blog about it within 15 minutes

by Administrator 28. March 2012 03:56

Source code: http://SolrContrib.CodePlex.com  change set: 96985

I had 15 minutes before I had to get ready to leave for work - I thought "would be really nice to have a [Launch Solr] button now that the "Launch Jetty" server button is working.   I thought - "you can do it" and was pleasantly surprised to find that not only was I able to plug this new feature in but I even have time left over (10 minutes) to blog about it.

Below is a screen shot of the SolrContrib desktop application.  I launched Jetty (console window) and clicked the "Launched Solr" button to load a browser window with Solr up and running. 

Outside of adding my constants to the Ctrl and Phrase files (see SolrContrib documentation.chm for more on these) the following is all of the code I had to write.  In the top pane I added a new LaunchBrowerCommand to the SolrMaintenance.Desktop\Commands folder.  Added a button the button to XAML (middle pane) in my SolrMaintenance.Desktop\Views\ToolboxView.xaml, and finally updated my SolrMaintenance.Desktop\SolrMaintenanceModule.cs file (bottom pane) to wire-up the button Tag to the new command.  Walla!   It is time to go to work and I have my button functionality plugged in :) 

Sure this is a small feat - the key point here is that this is a completely decoupled command which means, among other things,  that if I wanted to reuse this button on a different view anywhere within this module it would only take one line of code, all I would have to do is add the Tag (middle pane) to my new button.

 

Tags:

SolrContrib

Getting data in a decoupled environment - accessing ViewModel data from Command

by Administrator 19. March 2012 16:49

Accessing Data In Decoupled Framework

Source code: http://SolrContrib.CodePlex.com change set: 96837

As suggested in my blog on the Button Click Handler, developers for most applications cannot tell at a glance where code might be located for a given button click (particularly for an application they are not familiar with).   With the mindset that we want our framework for the http://SolrContrib.CodePlex.com project to be maintainable, we have a single point (the Presenter) where developers can go to find code (without having to have any knowledge of the visual tree).  


Figure 1.

If a Button Factory is used in a Presenter to process button clicks, the Module will hold the class name registered to the button (Tag property).   With that information the above [Execute Query] button can quickly be isolated to the following ExecuteQueryCommand (once again with no knowledge of the visual tree which designers could change).  Figure 2 shows all of the code required to achieve the above functionality (the framework handles the rest).


Figure 2.

Note above we have no reference to any class; we are completely decoupled.  We reference IPresenter (which could be any Presenter) to gain access to the ViewModel.GetViewModelData(sender,e) method.   This is available because IViewModel requires this method to be implemented as highlighted in figure 3.   Note that this method requires two parameters, object sender and EventArgs e, this is an extensibility point so that if in the future we do have to pass in parameters we won't have to touch any code that may have dependencies on it.


Figure 3.

The MvpVmViewModel (figure 4), which all ViewModels derive from, has default code for handling this process - it can be overridden if required.   By default, the process will populate a dictionary of all properties that have the custom SharedViewModelDataAttribute (ref line 153 of figure 4).   Note: we had to create extension methods for the Metro framework since this code is shared by both Metro and Desktop applications.


Figure 4.

The ViewModel for the SolrMaintenance module uses this attribute on its QueryViewInput property (line 24 of figure 5).  This allows the developer to choose which properties will be exposed to external processes; any process with access to the Presenter (IPresenter) will be able to access applicable values.


Figure 5.

Since the framework is a Multi Targeted framework, sharing code between Metro and Desktop applications - we can use the same functionality in the Gwn.DataBinding.Metro application that accompanies the http://SolrContrib.CodePlex.com project  (ref arrow in figure 6).


Figure 6.

The Presenter for Scenario #3 taps into the ViewModelPropertyChangedHandler (line 113 in image below) so that it will be notified when a property changes, it will then update the status bar (line 133 of figure 7).  It uses the very same process used by the Desktop application (line 129 of figure 7) to get the first property in the collection - in this case we have only one and it is the Name property (line 24 of figure 7). 


Figure 7.

ButtonFactory - Tag holds name of Command to process

by Administrator 17. March 2012 19:35

Button Factory

Source code is available at http://SolrContrib.CodePlex.com change set: 96810
The following video clip is discussed in this blog:

One of the goals of the http://SolrContrib.CodePlex.com project is to make an enterprise framework that will maximize reuse, be easy to extend and be maintainable.  We also want to be able to have an increased velocity without sacrificing quality - this will all be possible with the vision we hold for this project (see Vision Statement).

So we'll want to quickly add a new button to our framework and have it execute a command in a decoupled manner; the command executed should have its business logic completely encapsulated within the command (permitting reuse) - this is Patterns and Practices (P&P) 101. 

Ideally the wire-up should all be handled by an external configuration file which I hope will be possible when Unity is released for Metro, currently we're using a home-grown container.  For now I have to manually code three components to achieve this goal.  I have to tell my button Tag which Command to execute (on line 17 in Figure 1), in this scenario it is set to the constant ViewControls.ExecuteQuery (actual name is insignificant).  Now I have to register my command with the Unity container so that the ButtonFactory can resolve the Button.Tag for us - I do this on line 41  of the SolrMaintenanceModule. 

Figure 1.

I have setup my Presenter so that it knows to use the ButtonFactory (resides in Gwn.Library.xxxx where xxxx is Metro or Desktop).  I could have loaded down my presenter with all of the code using a case statement and/or methods for each process but this could quickly bloat my Presenter; for this module I will go with the factory pattern.

The video clip shows what happens when a button is not properly configured for this module - it crashes with an error that provides a hint to the problem.   If all is configured as expected we can call the factory so that it can get our concrete class.  In this case we're expecting an instance of ExecuteQueryCommand as configured on line 41 in figure 1. 

In figure 2 line 42 we'll tell the factory to process the applicable command, passing in a reference to the Presenter as well as the sender (button reference).  Since the Presenter is the hub of MVP-VM it will have a reference to the ViewModel, View as well as other components (such as business logic layers not currently used).  Since our ButtonFactory resides in the Gwn.Library project we'll have to create interfaces for anything we require access to, e.g., the SetResultViewOutput implements IResultViewOutput which will be utilized if the Presenter implements it (as this one does on line 16 in figure 2); this allows us to reuse the factory without tightly coupling it to this Presenter, ViewModel, or any other component. 

Note: Presenter business logic is done in regards to button commands - no further coding required.

Figure 2.

All that remains is to create my command which I store in the SolrMaintenance.Desktop\Commands folder.  As of this change set there are two commands - LaunchJettyCommand and ExecuteQueryCommand (a scaled down version of what will be the QueryAnalyzer).  Both commands essentially contain the following code (as of this change set).

Our command is not coupled to the Presenter or ViewModel - even though it is specific to this application.  One day I may need to reuse this command outside of the shared namespace/project, when I do if the Presenter passed in does not implement IResultViewOutput (line 25) then it will be null and will continue to work without crashing the business logic of the command.  We code for extensibility by coding to interfaces and not referencing classes which would be very easy to do in this command (via casting) to gain access to the Presenter and ViewModel directly.

 

using System;
using Gwn.Library.Factory;
using Microsoft.Practices.Unity;

namespace SolrMaintenance.Commands
{
    /// <summary>
    /// Launch Jetty Command
    /// </summary>
    public class ExecuteQueryCommand : ConcreteCommandBase
    {
        private IResultViewOutput _presenter;

        /// <summary>
        /// Constructor
        /// </summary>
        /// <param name="container"></param>
        public ExecuteQueryCommand(IUnityContainer container) : base(container) { }

        /// <summary>
        /// Execute process
        /// </summary>
        public override void Execute()
        {
            _presenter = Presenter as IResultViewOutput;

            SetResultViewOutput("Called from ExecuteQueryCommand!!\r\n{0}: Executing {1} button click!\r\n\r\n", 
                DateTime.Now, Parameter);           
        }


        private void SetResultViewOutput(string message, params object[] para)
        {
            // Only supports output if command implements IResultViewOutput
            if (_presenter != null)
            {
                _presenter.SetResultViewOutput(message, para);
                _presenter.PublishStatusBarMessage(message, para);
            }
        }
    }
}

Since we are decoupled and encapsulated we can easily unit test and when our work is completed will not have affected any other process - QA will only have to test this feature.

All roads lead to Rome - the ButtonClickHandler

by Administrator 17. March 2012 10:44

Button Click Handler

See following video clip for functionality discussed in this blog.  
Source code available at http://SolrContrib.CodePlex.com Change Set: 96808

I take a rather pragmatic approach to development, ironically to comply with the best practices and patterns (P&P) for architectural design (reference http://apparchguide.codeplex.com/).  For example "they" university would tell us that we shouldn't bind all of our buttons to one Command, i.e., every button should have its own command and be wired up to its own method (at some location such as a viewmodel or presenter).  This practice puts our pattern (and mind) in a clearly defined box affecting our maintainability and extensibility.  What would happen if we broke out of the box and applied rules defined in the P&P guidelines regardless of what "they" university would teach us?

You be the judge...  As for me, I'm writing a framework with the mindset that I will have to maintain it for the rest of my life! (my wife tells me I will "never" retire)

Here is a question - looking at the application shown in the video clip, how many views are there and what views contain the buttons?  Why would we programmers need to know this?  So that we can access the buttons business logic.  So where is the [Launch Jetty] button handled and where is the [Execute Query] button handled?  How could you possible know with this application (or any other application) until you dig in, sometimes deep depending on how creative the programmer was.  What if I told you that in five minutes you will "always" be able to answer this question with my framework - regardless how deep a hierarchy of controls are?   Come back to this question in a few...

Please reference figure 1 for the following discussion:

With the first draft of the framework completed (in an Agile sense) I am now able to focus on design and development.  So far I am pleased with the speed and efficiency in which I can code; I'm not being biased, all of the functionality shown in the video clip for the SolrMaintenance module was done with 38 lines of code.   I was able to create the module for my SolrContrib application within a few hours (would have been sooner but I had to deal with, and report a few bugs BUG1, BUG2, BUG3) - all of the functional code up to this point resides in the SolrMaintenancePresenter.  I have a focal point, the ButtonClickHandler, which will be common to any and every module I write.  This will help me (and developers new to my application) know exactly where to go when a button is clicked - regardless of where it is located in the visual tree.

The application shown below has three regions (MenuRegion, MainRegion, and StatusRegion).  Within the MainRegion, the SolrMaintenanceView (declared on line 15) has four views within it.   The [Launch Jetty] button resides in the ToolboxView while the [Execute Query] button resides in the QueryView (reference Figure 2).  Although this is nice to know, I can forget about "having" to know anything about the visual tree (which designers can freely change) while working with my infrastructure. Why? Because my http://SolrContrib.CodePlex.com enterprise framework has a single GenericCommand (in a viewmodel baseclass) that simply raises an event to notify subscribers a button was clicked; thus I have a decoupled process that components (status bar and application module) can hook into.   There are local events which are local to the current MVP-VM triad and global events that other modules (such as the status bar module) can hook into.

I should note that none of the views have code and (in the case of this module) are using the default viewmodel (SolrMaintenanceViewModel) declared on line 15.   Later when this application grows into a monster with many more modules, views and viewmodels a developer only need concern themselves with one notion - what module am I working on because it's presenter is where the code will reside (P&P extensibility and maintainability).

Read THIS BLOG to see the presenter utilize a ButtonFactory to process clicks

Figure 1.

Using a single GenericCommand (and event aggregation) means that a designer can easily move a modules buttons between views without affecting the modules code, regardless of hierarchy and its place in the visual tree!

Figure 2.