View Full Paper

Abstract

Reflexive-Transitive Closure (RTC) tables, when used to persist hierarchical data, present many advantages over alternative persistence strategies such as path enumeration. Such advantages include, but are not limited to, referential integrity and a simple structure for data queries. However, RTC tables grow exponentially and present a concern with scaling both operational performance of the RTC table and volume of the RTC data. Discovering these practical performance boundaries involves understanding the growth patterns of reflexive and transitive binary relations and observing sample data for large hierarchical models.

Contents

1 Introduction
2 Reflexive-Transitive Closure Properties
2.1 Reflexivity
2.2 Transitivity
2.3 Closure
3 Scalability Analysis
3.1 Rate of Growth
3.1.1 Depth-Wise Growth
3.1.2 Breadthwise Growth
3.1.3 Assessing Practical Growth Constraints
3.2 Time per Depth-Wise Growth
3.3 Space Consumption per RTC Table Growth
4 Conclusion
5 Appendix
5.1 Sampling Environment Specifications
7 References

I’ve walked through persisting hierarchical data via closure tables and then through abstracting the closure table in the application layer.  Now, we’ll look at rendering our hierarchy in an ASP.NET MVC 3 sample application using Razor. For the styling of the hierarchy I use jquery treeview which I quite like for its simplicity.  The typical approach to drawing tree views / hierarchies in the view is nested html unordered lists (<ul>).  To traverse and render our Hierarchy of family members we’ll leverage the power of the Razor view engine to create a set of reusable partial views for drawing our hierarchy whenever we need it.

When its all said and done, we’ll end up with:

Well, that looks simple enough, let’s see how we get there!

Hierarchy.cshtml

@model Hierarchy
@{
    ViewBag.Title = "Hierarchy";

    //Create the PartModel for the Hierarchy in preparation for the ViewPart
    //Note we are passing in the name of the ViewPart we want to use
    //This allows the rendering of the node to be defined inependently
    //of the structural rendering of the hierarchy
     var partModel = new HierarchyPartModel()
                    {
                        Hierarchy = Model,
                        HierarchyNodeViewPart = "Parts.Home.FamilyMemberNode"
                    };
  }

@if (Model != null)
{
    //Call upon our Hierarchy ViewPart to handle the rendering
    //of the hierarchy structure, passing in our HierarchyPartModel
    //What we are accomplishing here is delegating out the logic
    //for rendering parts of the view to partial views. Hence, "ViewPart"
    @Html.Partial("Parts.Common.Hierarchy", partModel)
}
else
{
    @Html.RouteLink("Create Hierarchy", "AddNode", new { parentID = 0 })
}

Let’s take a moment to review what we just witnessed. First, it may be a little puzzling to find no trace of HTML in our view – what gives? Just one of the awesome aspects of Razor is the ease with which we can create layered views. Huh, sounds abstract, what does that -really- mean? Well, let’s reflect upon the more typical way of working with our view pages. Our controller would return a View (ActionResult) and when the view renders the HTML would exist entirely in the View at best some Editor/DisplayFor templates and some partial views (from Html.RenderPartial). What Razor allows us to do is grow beyond those more primitive template techniques and provide multiple Layouts and new ways to invoke partial views (Html.Partial), etc. My colleague and I began calling these partial views “ViewParts” and the View Models we pass in, “PartModels”. This approach was inspired by the work the Orchard team did with their approach using Razor. The benefit to this approach is a high degree of reusable HTML “controls/components” (ViewParts) and a clean distribution of view logic across the presentation layer. I admit, its awkward at first but after coding with it, I can’t think about it any other way.

Let’s peel back the layers now and see what’s going on in the ViewParts, starting with the PartModel we passed in.

HierarchyPartModel.cs

    //This is a logical equivalent to a ViewModel, except
    //specifically meant to support a ViewPart(s). The reason
    //for a derivative naming convention is to prevent confusion
    //with patterns that embrace ViewModels like MVVM
    public class HierarchyPartModel
    {
        //The core model we are working with
        public Hierarchy Hierarchy { get; set; }

        //This is the name of the ViewPart to use for rendering
        //each node of the hierarchy, we will see how this is used
        //in the ViewPart this PartModel supports
        public String HierarchyNodeViewPart { get; set; }
    }

Parts.Common.Hierarchy.cshtml

@model HierarchyPartModel

@*
    //Another very cool feature Razor provides us, inline
    //HTML helpers. The primary responsibility of this helper
    //is to assist with drawing the nested unordered list
    //for our hierarchy
*@
@helper DisplayHierarchy(FamilyMemberNode startingNode)
{
    <ul>
        <li>
            @*
                //Here we are rendering another ViewPart, specifically
                //the name of the ViewPart defined in our PartModel
                //The idea here is that we are separating the responsibility
                //of rendering the node from the structure itself
            *@
            @Html.Partial(Model.HierarchyNodeViewPart, startingNode)

        @*
            //If our node has children begin drawing those branches of the hierarchy
        *@
        @if (startingNode.HasChildren())
        {
            <ul>
                @foreach (var child in startingNode.Children)
                {
                    @*
                        //RECURSIVE CASE:
                        //Here we begin our logic all over again with
                        //drawing the node and nesting unordered lists
                    *@
                    @DisplayHierarchy((FamilyMemberNode)child)
                }
            </ul>
        }
        </li>
    </ul>
}

@*
    //Define our container for which jquery treeview will target
*@
<div id="tree">
    @*
        //Invoke the initial call to our helper
        //passing in the top-most root of our hierarchy
    *@
    @DisplayHierarchy(Model.Hierarchy.GetRoot())
</div>

Parts.Home.FamilyMemberNode.cshtml

@model FamilyMemberNode
@*
    //HTML definition for each node in the hierarchy
*@          
<div class="hierarchyNode">
    <div>@Model.FamilyMember.Name</div>
    @Html.ActionLink("Add", "Add", "Home", new { parentID = Model.FamilyMember.FamilyMemberID }, null)
</div>

So, quick recap. Our controller returns an ActionResult and when Hierarchy.cshtml is rendered it defines a HierarchyPartView shoving in the Hierarchy data structure and the name of the ViewPart to use as the HTML definition of each node. Then, Hierarchy.cshtml calls upon Parts.Common.Hierarchy.cshtml to handle rendering the structure of the hierarchy and it does so by leveraging a recursive inline html helper. From here, the ViewPart defined in the HierarchyPartModel is used to to render each node.

I wouldn’t be offended if at first you thought to yourself… “Hey buddy – way to over-engineer the hell out of your view, now I’ve got to inherit this onion”. On the surface, it may very well appear that way. However, let’s look at what we have gained with this implementation approach. We’re able to render a hierarchy anywhere we wish. I don’t argue this wasn’t possible in MVC 2 or even WebForms, but I do argue that Razor has made this approach FAR more manageable than prior view tech. Also, while I was implementing this approach on a project it was not decided which technology to use for styling the tree-view. This approach was validated when asked to change the various aspects of the tree. It proved to be very painless to quickly swap out the JQuery treeview approach for Telerik MVC Treeview. My nodes didn’t change, and neither did my Hierarchy.cshtml page. Also, the nodes needed to be defined differently depending on where the hierarchy was rendered in the view (and an array of other “business rules”). No problem, create a new ViewPart for defining the node and pass it in. Gone are the days of copy/pasting/fitting HTML across many views and feeling the burn of managing change in the presentation layer.

I’ve also used the technique with jstree, telerik mvc treeview, and even some cool open source css like Slickmap.  Hey, B^2 – what’s the point?  Right, the point is we’ve done a great job persisting and generating our hierarchy and we have total freedom in the UI to display our hierarchy with a high degree of reuse and flexibility.

Hopefully this series of working with Hierarchical Data has been useful – I know I learned a lot during discovery and implementation phases of these concepts.

Feel free to download and use any of the code from this sample project: DaHierarchy

In my previous post I walked through persisting hierarchical data using closure tables.  While closure tables are a great technique for persistence we need a more approachable way to deal with the hierarchy in the application layer.  Since we are dealing with a hierarchy it stands to reason that we could benefit from a tree-like data structure and that’s just what we’re going to construct.

    //I tend to favor interfaces over abstract classes
    //However, this is definitely one of those places where
    //either/both make sense. The point of the node is to
    //represent a single record within the closure table
    //This is where the abstraction begins
    public interface IHierarchyNode
    {
        //Here are the three fields from the closure table
        Int32 ParentID { get; set; }
        Int32 ChildID { get; set; }
        Int32 PathLength { get; set; }

        //Since we are not limiting the width of the hierarchy
        //at any branch we don't have a typical "left child"
        //"right child" approach instead we prepare for n children
        IEnumerable Children { get; set; }

        //Useful convenience method
        Boolean HasChildren();
    }

The node itself is pretty straight forward.  My first implementation of the HierarchyNode used generics and was quite flexible.  Unfortunately, the solution I was working in used Entity Framework 4.1 Code First(not the unfortunate part) and as it turns out EF 4.1 doesn’t support building models which implement generics (the unfortunate part).  So, I backed up to a slightly cumbersome approach.  The takeaway here is that it doesn’t really matter what patterns you apply to this scenario, at the end of the day we just need to get a closure record out of persistence and represented in the application.

    public class FamilyMemberNode : IHierarchyNode
    {
        //Implementation of IHierarchyNode - nothing to see here
        public Int32 ParentID { get; set; }
        public Int32 ChildID { get; set; }
        public Int32 PathLength { get; set; }

        public IEnumerable Children { get; set; }

        //The major difference here is that our FamilyMemberNode
        //carries a payload of a FamilyMember.  Again, we could
        //avoid concrete classes of the node and use generics
        //or other techniques, but lets not get hung up on that
        //conversation
        public FamilyMember FamilyMember { get; set; }

        //Covers our basis for children. There is a very compelling
        //argument to move this method to an abstract class to cut
        //down on DRY violation.
        public bool HasChildren()
        {
            if (Children == null)
                return false;

            if (!Children.Any())
                return false;

            return true;
        }

        //Providing a way to compare two nodes is invaluable
        //especially for unit testing and tree traversal.
        //Another candidate for abstract class
        public override Boolean Equals(object obj)
        {
            var node = (IHierarchyNode)obj;

            //Two nodes are equaly if all three of these fields are identical
            return (ParentID == node.ParentID &&
                    ChildID == node.ChildID &&
                    PathLength == node.PathLength);
        }
    }

With the nodes out of the way, we can move on to building the hierarchy and providing an abstraction from the closure table. Prior to digging in, it may be worth having a look at the closure table results to follow along with some of the logic here. The nomenclature becomes a little challenging to navigate here. At one point I refer to the ROOT of the entire hierarchy, and other times I mean a root/parent/node to be a record in the closure table where the ParentID = ChildID and PathLength = 0. Hopefully I’ve landed on something that gets the idea across.

    public class Hierarchy where T : class, IHierarchyNode
    {
        //Think of this as the "Genetic Code" of the hierarchy, we hold
        //the metadata about the structure here.  Effectively, we hold
        //the closure table records and can query these nodes just as
        //we did in SQL against the closure table
        private readonly IEnumerable _unstructuredNodes;

        //Top most node of the entire data structure.
        private T _root;

        public Hierarchy(IEnumerable nodes)
        {
            _unstructuredNodes = nodes;

            //Builds the entire hierarchy when instantiated
            Build();
        }

        ///<summary> 
        /// Retrieves the root of a given node. This root may not be the root of the entire
        /// hierarchy, but instead is logically equivalent to the parent of the given node.
        /// Since the underlying structure of nodes may suggest there is more than one parent
        /// this method finds the actual root (Where PathLength = 0)
        /// </summary>
        public T GetRoot(Int32 nodeID)
        {
            var root = _unstructuredNodes.Single(x => x.ParentID == nodeID && x.PathLength == 0);
            return root;
        }

        ///<summary>
        /// Returns the root HierarchyNode of the entire Hierarchy structure.
        /// </summary>
        public T GetRoot()
        {
            //If we already know the root, don't figure it out again
            if (_root == null)
            {
                //The logic behind selecting the root is as follows:
                //Root(or parent) nodes are those where PathLength == 0
                //The Root of the entire structure is the parent node that is not the child of
                //any other parent nodes.  Below, the &&(AND) clause is executing the statement
                //"where the parent node is not a child of any other node"
                var root = _unstructuredNodes.Single(x => x.PathLength == 0
                                            && !_unstructuredNodes
                                                .Where(t => t.PathLength != 0).ToList()
                                                .ConvertAll(t => t.ChildID)
                                                .Contains(x.ParentID));

                return root;
            }

            return _root;
        }

        //Builds the entire hierarchy starting from the top-most root
        private void Build()
        {
            _root = GetRoot();
            _root.Children = BuildChildNodes(_root);
        }

        ///<summary>
        /// Recursively constructs the child nodes from any given node down.
        /// That is, given any node, this routine will construct the entire
        /// branch of the hierarchy starting with provided node as its root
        /// </summary>
        public List BuildChildNodes(T node)
        {
            //These are the children that will be constructed and
            //ultimately become the given node's Children field
            var parentNodes = new List();

            //These are the immediate children of the given node, but are not "Root" nodes themselves,
            //that is, the root node is such that PathLength = 0 and subsequently ParentID == ChildID
            var children = _unstructuredNodes.Where(x => x.ParentID == node.ParentID && 
                                                         x.PathLength == 1).ToList();

            //We iterate through these children to construct the appropriate set of nodes that will
            //become the given node's Children
            foreach (var child in children)
            {
                //Get the root node of this child
                var parentNode = GetRoot(child.ChildID);
                parentNodes.Add(parentNode);

                //RECURSIVE CASE:
                //Now we go find any children of this node until we find a leaf
                BuildChildNodes(parentNode);
            }

            //BASE CASE:
            //When this point is reached, we either have no more children to iterate through
            //or a leaf was found and children.Count() == 0, so return given node with its children
            node.Children = parentNodes;
            return parentNodes;
        }
    }

The data structure above is certainly not complete but I’ve accomplished what I set out to do – abstract the particulars of the closure table behind a data structure within the application. At this point, the rest of the functionality is left up to individual style. For example, traversing, searching and CRUD operations could be implemented with a fluent interface, or a set of extension methods on the base Hierarchy class without actually pushing the operations into the base class itself. I may offer an approach in a future post after coding with it a bit more.

My objective with the Hierarchy data structure above achieves the abstraction I was looking for, and in the next post we’ll work with the hierarchy by rendering a UI. However, I want to touch on a few points before moving on. The hierarchy above has some definite scaling issues. When I created this data structure, I was anticipating a reasonably small size hierarchy of approx. 100-200 nodes in the hierarchy and a bit over 1000 unstructured nodes (entries in the closure table).

The first pressure point for scaling is my use of recursion, and we are all likely aware of recursion’s performance woes. Should you find the hierarchy performing poorly perhaps re-implement the construction of the hierarchy with an iterative approach. The second pressure point for scaling is holding on to the unstructured nodes (all closure table records). It may be convenient in the short term to query the list via linq like we would the closure table using SQL. However, keep in mind that the hierarchy itself only holds a relatively small percentage of the unstructured nodes from the closure table. Releasing the list of unstructured nodes may greatly reduce the memory footprint should you find yourself working with enormous hierarchies.

Hopefully this helps get your head around working with the closure table and serves as an alternative to heavy stored procedures in the persistence layer.

Feel free to download and use any of the code from this sample project: DaHierarchy

Recently, I’ve been working with hierarchical data using closure tables.  The problem this technique solves for me is persisting hierarchical relationships of specific entities without any restriction on the size of the hierarchy and while providing a simple way to query the hierarchy.  Closure tables, above other solutions like path enumeration, maintain referential integrity.

In a series of posts about storing hierarchical data I’m going to walk through implementing a solution for working with hierarchical data at the persistence layer, the application layer, and the presentation layer.  We’ll see what is necessary to persist the hierarchy in SQL Server 2008, how we wrap the closure tables in the application, and how we build hierarchy in the user interface with MVC 3

First, let’s dig right in and look at what’s necessary to set-up the closure table.  In the image below you’ll see just what little is necessary to get going.  The table titled “Closure” is the closure table where the metadata about the structure of the hierarchy is stored.  The ParentID is the foreign key to the FamilyMemberID and the ChildID is the FamilyMemberID of one of the decedents.  The PathLength indicates how far removed from the node the “Child” is.  So, if we are looking at a grandfather, then themself would be PathLength = 0, their child would be PathLength = 1, and their grandchild would be PathLength = 2.

You may have also noticed that the FamilyMembers table also holds a ParentID.  This makes things convenient, but is not completely necessary.  To understand more about why the ParentID is present, let’s look at the trigger “FamilyMemberClosureTrigger”.  Everyone’s comfort level with database triggers is different but to me this is one of those appropriate cases for one.  In this case, the trigger is responsible for creating the metadata entries for the Closure table each time a record is inserted into FamilyMembers (or updated, deleted, etc).  This isn’t absolutely necessary, though.  You could definitely move this logic elsewhere and do away with the trigger altogether.

--Here, we create the trigger when an INSERT takes place in FamilyMembers table.
--Update and Delete are still necessary but left them out for this demo
CREATE TRIGGER [dbo].[FamilyMemberClosureTrigger] ON [dbo].[FamilyMembers]
FOR INSERT
AS
 DECLARE @MemberID INT;
 DECLARE @ParentID INT;

--Get the FamilyMemberID of the newly inserted record
SELECT @MemberID = i.FamilyMemberID FROM INSERTED i;
--Get the ID of the new record's parent
 SELECT @ParentID = i.ParentID FROM INSERTED i;

--First, we insert the record for the newly inserted FamilyMember
--This is a self-referencing record, meaning the ParentID = ChildID
--And PathLength = 0.
INSERT INTO Closure(ParentID, ChildID,PathLength)
 VALUES(@MemberID, @MemberID, 0)
--Next, we insert metadata about how this new family member
--is related to the rest of the hierarchy.
INSERT INTO Closure(ParentID, ChildID, PathLength)
SELECT p.ParentID, c.ChildID, p.PathLength + c.PathLength+1
 FROM Closure p, Closure c
WHERE p.ChildID = @ParentId
AND c.ChildID = @MemberID

At this point, we’re all set-up for new FamilyMembers to be inserted into the FamilyMembers table. Let’s take a look at the data when these tables are populated.  Below, you can see the Closure Table (left) and the FamilyMembers table.  Take a moment to observe just what the Closure Table is storing for us.

The Closure Table, by storing this metadata about the FamilyMembers enables us to directly query the relationships in the hierarchy pretty quickly.  That is, we can ask questions like “Who are all of the grandchildren of John Doe”

SELECT * FROM Closure c
WHERE c.ParentID = 1
AND PathLength = 2

Which results in:

As you can see, this makes things relatively simple for querying. Though, there are some downsides to this solution. We can see the size of the closure table will grow exponentially faster than the entity table (FamilyMembers). Also, the closure table is not simple to “debug”, that is if the table becomes corrupt i.e. failed writes, etc, then the table will need to be rebuilt as attempting to track down the broken relationship visually is not impossible but an exercise in tedium. On the other hand, the relationships are pretty simple so rebuilding part or all of the hierarchy isn’t so bad, especially if you also implement the ParentID on the entity table itself – its part of the charm.

In the next post I will be walking through how to build a custom data structure to abstract the application from the closure table and work with the hierarchy throughout the application.

For anyone who loves their work its tough to admit its time for a vacation, let alone actually take one. I’ve always had a strong work ethic even when it came to work I didn’t like very much; when I do enjoy the work I’m doing you can replace “work ethic” with “work-a-holic”. When it comes to software engineering its an obsession. I’m always tinkering with something either at home or the office. I know full well I’m not alone. Most coders I know who love their profession are always coding either on the keyboard or in their head.

The downside to loving what you do is not knowing when its time to take a break. Recently I took vacation and made some observations about the various states I was in throughout the week. Its definitely tough to unplug from the cool work we do and how can we complain, right? We get paid to tell computers what to do all day, which is really cool! However, nerds need a break too and its not wrong to admit to burn out.

1. State of Denial: I don’t really need a vacation

First, there is the state of denial about needing a vacation. I knew a lot of family would be in town for a week and I also knew a lot of deliverables were due the week after they left. Immediately, I told myself I don’t need a vacation and I’ll just work remote – that’s the same as being home while family is there, right? It took a lot to admit to myself that this is crazy. Being home and working on client deliverables doesn’t mean I’m home; it means I’m shut in my office with my heads phones on and if family wants to see the back of my head they can open the door and even converse with me without my knowing it.

We all know the addiction to deliverables and coding through the best and worst of them. We also know that when deliverables dates are met, a new set of deliverables are scheduled. So, there isn’t really a natural break in deliverables, all there is to do is trust in the rest of the team to handle the deliverables while you’re on vacation. For me, this is the only way to transition out of the state of denial; admit the vacation is needed and trust that life will go on without you. Its harder than it sounds.

2. State of Withdraw: Keep your coffee and bagel, where’s my laptop?!?

The first morning of vacation is awkward. I can imagine it happening one of two general ways for the truly addicted developer, I experienced both. Either you will roll out of bed and your feet will hit the floor in stride toward your laptop or you’ll stumble through canned morning greetings — “morning, how’d you sleep?” … “oh you know, with my eyes closed” — and shuffle right passed coffee and bagels for your laptop. The guilt set in pretty much immediately when I started checking up on tickets to see what’s moved and what code has been committed. If you’re lucky, you’ll have colleagues that will keep pushing you away from work and family who will keep shoving coffee and bagels in your face and giving you queues about what humans do for breakfast (Hint: It doesn’t involve your computer). Its your support group or sheer will power that transitions you out of this state.

3. State of Detachment: Man, I bet there was some cool cube conversation today…

Detachment comes in a few forms, and it closely follows withdraw. For me, I not only love my profession but the environment I work in is awesome so it didn’t take long to start wondering what I was missing at the office. I wondered if we got the new client I heard about? What cool conversations was I missing? These are the kinds of things I started to think about. Its easier to transition out of this state if your feeling detached from a positive environment. Getting out of the house and getting busy helps a lot, as does playing drinking games where the loser plays a round of “Just Dance” for the Wii.

There is a darker side of detachment, though, and while this isn’t a product of my current environment I’ve felt this way about places I’ve worked in the past – places I wasn’t very happy to work at. Feeling detachment can come in the form of worry or stress: “I wonder if anything is going wrong at work”, “What if something happens and gets blamed on me?”, “I bet they are talking trash about me since I’m away”, “Maybe my supervisor thinks I’m weak for taking time off”. If you feel this way and you’re a developer consider looking for a better place to work. If you’re not a developer and feel this way, consider looking for a better place to work. It shouldn’t be wrong to take vacation and if it feels that way then getting out of this state may not happen for the rest of your vacation.

4. State of Rest: Time for all the code I’ve been meaning to write!

I reached a point in my vacation where I was feeling great, the hangovers where light, the sleep was restful, I had forgotten about work and the rest of the world and family had a great time before heading home. Wouldn’t it be cool to relax with some code – after all, it can be relaxing to do the things you like to do, right? We all have those cool ideas and projects we want write code for but can’t because of how busy work (or life) is. Its risky business to mix vacation with code and its going to be tempting to make excuses like “well there isn’t anything else to do” or “I’ve wanted to do this for a long time and I just never get the chance!”. For me there is a lot of truth to these arguments but the risk here is not fulling decompressing from the mindset of problem solving and code. Feeling rested isn’t necessarily a sign that its time to jump back into the fray. Transitioning out of this state, for me, required more rest and more time out of my the engineering headspace.

5. State of Clarity: “Moisture is the essence of wetness, and wetness is the essence of beauty.” ~Zoolander

Vacation was a success when I felt clarity again. What I mean is when I think about work again I don’t feel pressure about the deliverables or the amount of work ahead of me. Instead, I just know what needs to be done and feel like tackling the work isn’t a big deal. I know I do my best work when I can anticipate the problem being solved, the risks associated with it, the changes that are likely to be requested, etc. When I can think like an engineer again, with clarity, I know its time to return from vacation. This state of clarity may seem like a subtle difference from the State of Rest, but being well rested is just not enough to make a full return from vacation. In fact, its reaching this clarity that opened my eyes to how I’ve done this whole “vacation” think wrong in the past, where I returned to work too soon or fooled myself into thinking that working half the time I’m on vacation is a vacation.

I’ve gained perspective on the need to take vacation and how to work through a vacation in a way that puts me back at work better than when I left. While we, as a profession, have some very cool jobs and work with awesome technology it takes its toll on us and we need to decompress. So if you’re currently in your cube shaking your head at this post and thinking you couldn’t possibly take vacation then its time you take one!

If passion drives you, let reason hold the reins. 
Benjamin Franklin 

Given that I work with fellow passionate and intelligent nerds (a deadly combination) there is always a level of intensity around the office – it would be an awfully boring environment where that not the case.  

I’ve come to a realization just today that not keeping up with the level of intensity of those around me is not a sign of weakness – or an inability to “keep up”.  In my particular case, its a sign of total denial about what I’m passionate about.  

Looking back, and likely to those reading this, my mistake is pretty obvious.  The level of intensity people have when discussing a topic is due to their passion for the topic.  Attempting to match that intensity means I’d have to also be passionate about the topic.  Being passionate about a topic means I must then pour lots of time into keeping up on said topic – that’s what passionate people do with the things they are passionate about, right?  Right – of course!

Attempting to keep pace with someone else’s intensity and passion because I want to “get it” too and because I don’t want to “fall behind” is totally crazy – learn from my mistake!  As it turns out, its far less stressful to simply accept what I’m passionate about and leave others to own what they are passionate about.  In fact, it leads to actually enjoying other’s enthusiasm for a topic. Already, I’ve learned more from conversations today – I suspect that’s because I’m listening more.

I attempted to start the Storage Emulator from: Start Menu -> Windows Azure SDK v1.4 -> Storage Emulator. The Azure SDK command prompt opens and informs me its attempting to start the Storage Emulator but a red text message flashes on the screen and the command window closes. 

Getting to the bottom of this is not a big deal, in fact its worth considering just using the Azure SDK Command Prompt instead of the above “convenient” controls.

Open the Azure SDK Command Prompt from the Windows Azure SDK v1.4 directory (or Start Menu). csrun.exe is found in the bin directory.  Change directories then execute csrun /devstore:start.

After several moments, we finally get an error message that sticks around long enough to give us a clue.  As it turns out, the message is quite useful even!  In my case, I develop in a virtual machine and so I tend to keep things off if I’m not using them.  So, I only keep SQL Server R2 running, and leave EXPRESS and 10.0 off.  

The fix is easy enough, open up SQL Server Configuration Manager and start SQL Server Express.  Once the SQL Instance is running, starting the Storage Emulator by any of the methods mentioned prior will work without a hitch.

Well, hope that was enlightening.  Its pretty easy to work directly with Azure SDK via the Command Prompt.

1. Create Analysis Services Project
2. Create a Data Source
3. Define a Data Source View
4. Create Data Cube and Dimensions

Set-up

First things first, open up BIDS by going to start menu, Microsoft SQL Server 2008 and clicking on SQL Server Business Intelligence Development Studio (BIDS).  With BIDS now open create a new project by selecting from the FILE menu a new project (File -> New -> Project…) to get the following dialogue.

Select “Analysis Services Project” from the Templates window then move on to name the project and select a location for the solution directory to be created.  In my case, I have named the project IISLogCube and left the default path in the Location field – which defaults to the Visual Studio 2008\Projects folder.  Clicking OK will create your solution and project directory and deliver you into the new project.  Pin the Solution Explorer window to the right and look at project structure created for you.

The folders we are interested in for now are the first four. The good news is creating the items for these folders is largely automated and/or wizard driven. Let’s get started by creating a Data Source.

Create a Data Source

Right click on “Data Sources” and select “New Data Source” to being the Data Source Wizard (below).

Click “New…” to discover and select the server and database. I will be using the Logs database we created for the IIS Logs Data Warehouse.  Select the name of your server from the drop-down list under “Server name:”.  If you’re running local either type in (localhost) or click Refresh to discover local server names.  Configure whatever server credentials are necessary under “Log on to the server”.  Once the server configurations are complete move down to “Connect to a database” and select the database from the drop-down list.  Optionally, you can click “Test Connection” to make sure all the settings are good to go before clicking OK.

Once you’re connection is configured you’ll notice the new Data Connection option back on the Wizard window. Select the Data Connection you just configured and select “Next”.

The Impersonation Information is needed to connect to the data source.  If you are unsure what to select here, choose Inherit (named Default in BIDS 2005).  You will know if you chose correctly when you process the cube, which we’ll do later.  We can come back and change this option at any time, so don’t fret too much about this option for now.

From here, click “Next”, name your Data Source then select “Finish”.  You’ll now see your Data Source in the Solution Explorer under the “Data Sources” folder. Now we’ll need a Data Source View.

Define a Data Source View

Right click on Data Source Views and select “New Data Source View…”.  Select Data Source from the list of “Relational Data Sources” and select “Next”.

Name matching is meant to assist with how relationships are discovered in your data source.  It is helpful if the data source matches one of the conventions here but it is not required. The first option applies to this sample data source.  Select the appropriate option for your data source then click “Next”.

Now we’re getting to some of the more interesting configurations.  Here you’ll choose the tables and views from the database in your data source that you want available to your data cube.  If you select more tables than what you think you need, that’s OK as you don’t have to use them all in the Cube. That being said, its not a good idea to choose ALL available objects (unless you really need to) as that may be a considerable amount of tables and views and may slow down processing of your cube.  At a minimum, you’ll need to bring in a Fact table and a few tables that will act as dimensions. Click “Next”, name your view and click “Finish”.

Now that you have created your view, you may need to clean it up.  As you will notice, the relationships were not detected for Companies and Clock table as related to the LogFact table.  We can manually clean up these relationships now – and we’ll want to do so to get the most of our Cube creation step.

This is where some of the name matching we configured in the Data Source View wizard came into play.  You’ll see the relationship was detected and set-up for the UrlID between the LogFact and Urls table.  Other relationships were not detected, however, because the primary keys were not properly identified in our data source (intentionally, for this demonstration purpose).

We can set the record straight, so to speak, by right clicking on the primary key in each table here and selecting “Set Logical Primary Key”.  Take Users for example.  Right click on “UserID” on the Users table and choose “Set Logical Primary Key”.  You’ll see the key icon appear next to the “UserID” field.  Next, we’ll want to draw the relationship between the LogFact table and the Users table.  To do so let’s click on the “UserID” field on the LogFact table and choose “New Relationship…”.  The left shows the Foreign Key and will automatically be populated since we initiated this new relationship by clicking on the “UserID” field on the LogFact table. Set the Primary Key (the right side) by selecting the primary key table “Users” from the drop-down list then the field from the drop-down list below.

After defining logical primary keys and setting up appropriate relationships we get the following corrected View.  Some of the relationships may not make total sense just yet, but we’ll see why we have them when we review our relationships later.

Create Data Cube and Dimensions

Creating the Cube and Dimensions is automated for us by completing the Cube Wizard.  Right click on “Cubes” folder in the Solution Explorer and select “New Cube”.  As you’ve likely noticed by now the steps we have completed up until now have made things pretty simple and automated.  By selecting “Use existing tables” in the Cube Wizard we are able to create a cube from the tables we selected and set-up in the Data Source View step.

After selecting “Next” we are prompted to identify the table that defines the measurements, most commonly known as the fact table.  If you hadn’t followed the traditional “Fact Table” approach, then you can select “Suggest” and the wizard will attempt to determine which table(s) are potentially a fact table.  More than one Fact table may be selected, which is appropriate for Fact Constellation Schemas.

After identifying the fact table(s) select “Next”.  On the following screen you’ll identify (or confirm) the fields that are checked are measurements you want for the cube.  Selecting “Next” again completes the measures group portion of the Cube Wizard.

Following the measure groups portion, you’ll need to select the Dimensions you want included in the cube.  The wizard will provide a list of tables that are already selected, assuming that you want them as dimensions.  Selecting “Next” through the remainder of the wizard confirms the selection of Dimensions.  Lastly, provide a name for the Cube and click “Finish”.

Completing the Cube Wizard automatically creates the cube and dimensions.  The resulting Start Schema is produced for our IIS Logs.

Also, note the solution explorer now has Cubes and Dimensions represented in their respective folders.

Now that our cube is complete, we want to build and deploy.  To do so, right click on the Analysis Services Database “IISLogCube” and select “Process…”.  You’ll be prompted for confirmation that you want to build and deploy the project.  Select “Yes”. If you are not deploying to server (localhost) then you will likely receive an error since the default Server for the Analysis Services Database is localhost.

Right click on “IISLogsCube” and selection “Properties”  On the resulting window select “Deployment” and under section “Target” change the “Server” from localhost to the appropriate server name that you will be deploying this database to.

If there are no errors during deployment then you’ll be prompted with the pre-process information.  Clicking “Run…” will begin the processing of the cube. Clicking “Run…” shows the process progress window.  Once the processing is complete you can close the progress window and the “Process Database” window.

At this point, the cube has been successfully built and deployed.  While this has been a somewhat long description of the above three steps its largely due to a thorough look at the Wizards that help this process along.  It doesn’t take long before cubing data is a fast and fluid process – the Wizards in BIDS certainly does streamline the process.

There is a great deal of configuration and tweaking that may be applied to the cube.  I’ll go over thee in future posts, including creating Hierarchies.

The fact table “Log Fact” is largely comprised of IIS Log data as the measurements – specifically timetaken, scBytes, csBytes.  Normally, I may prefer a cleaner naming convention for my fact table but I thought it might be beneficial to keep the IIS Log naming convention so we can refer back to the IIS Log specification with less guess work involved.

The dimensions – Users, Urls, Companies – represent the context of the application for which these logs belong, such as company, user and office data. Keep in mind that this is all sample data I created and I didn’t go to such lengths as to create a fancy fictitious context.  In your applications the dimensions you may add for context would likely have more interesting data.

Speaking of application context, what is it? What I mean by application context is the application concepts (e.g. domain entities, events, etc) that add meaning or value to the IIS log entries.  In this sample, we have Users, Companies and Offices as our application context so we may know what users for what company (and potentially what office) are accessing pages at specific dates and times.  I have chosen to associate the IIS Logs to the application context based on IP Address. So, when considering warehousing your IIS Logs, you’ll want to make sure you have some way to associate your log data and application data.

When deciding to begin logging application related data more is not always better, and the same goes for deciding what IIS Log data to capture.  The more verbose your logging, the more data you must clean and maintain.  If you don’t need it, don’t log it.  A great way to manage data greed is to only capture information that helps answer the question(s) for the business – put another way, only collect the data that adds immediate value. This will save you time warehousing, cubing and otherwise crunching the data that is important – especially if you’re resource constrained (e.g. storage space or hardware).

Below are the tables I am working with.  Recall in the previous post I moved all of the IIS Logs into the Logs table via LogParser.  There are a few ways to go about loading data into the warehouse.

One way would be to edit our LogParser script to load directly into the LogFact table by making our SELECT statement.  The Pros is that we don’t need to store the data twice, once in a “raw” table and a second time in the fact table.  The cons is that you’re control of your ETL package is entirely dependent on LogParser, which arguably is not meant for ETL, but instead for querying log data via SQL. Also, LogParser won’t help with merging IIS Logs and application context so a second step or a stronger ETL strategy would be necessary.

In this case I loaded the data into the Logs table and then created a SQL ETL script to move select data out of the Logs table while merging it with my application context and loading the data into the warehouse schema. For huge volume situations the bigger toys like SSIS may be useful – which may even replace LogParser at that point.

Here is the ETL script I fashioned for this prototype:

--Gets most current IP list
SELECT DISTINCT IP, MAX([DateTime]) AS LastLogin
INTO #IPs
FROM LoginLogs
GROUP BY IP
ORDER BY IP DESC

--Gets Company and User IDs based on most current IP list (above)
SELECT DISTINCT ll.IP, o.CompanyID, ll.UserID
INTO #TempIPs
FROM LoginLogs ll
JOIN #IPs i ON (i.IP = ll.IP)
JOIN OfficeUsers ou ON (ou.UserID = ll.UserID)
JOIN Offices o ON (o.OfficeID = ou.OfficeID)
WHERE ll.[DateTime] >= i.LastLogin
ORDER BY IP DESC

INSERT INTO LogFact
SELECT
	l.LogID,
	l.cIp,
	l.csUriStem,
	l.[date],
	CAST(l.[time] AS time(0)) AS [time],
	DATEPART(hour, l.[time]) AS [hour],
	ROUND(l.timetaken, -2) AS timetaken,
	ROUND(l.scBytes, -2) AS scBytes,
	ROUND(l.csBytes, -2) AS csBytes,
	tmp.UserID,
	tmp.CompanyID
FROM #TempIPs tmp
JOIN Logs l ON (l.cIp = tmp.IP)
WHERE timetaken IS NOT NULL
AND LogID > COALESCE((SELECT MAX(LastProcessedLog) FROM LastProcessedLogs), 0)
GO

--Make an entry noting the last record processed so we don't process the data more than once
IF (SELECT MAX(LastProcessedLog) FROM LastProcessedLogs) < (SELECT MAX(LogID) FROM LogFact) OR (SELECT MAX(LastProcessedLog) FROM LastProcessedLogs) IS NULL
BEGIN
	INSERT INTO LastProcessedLogs SELECT MAX(LogID) as LastProcessedLog, GETDATE() as DateProcessed FROM LogFact
END
GO

DROP TABLE #TempIPs
DROP TABLE #IPs

In a future post, I will be discussing how to create a data cube from our new warehouse, which gets us a step closer to getting some real cool knowledge from our data!

Log Parser 2.2 is a command line tool which supports querying text-based data from many log file formats.  That is, familiar SQL-like statements can be executed against log files for fast access to the data within.  In the case of Log Parser the documentation included with the download is very reliable so much so that Google is likely not the first place to go with questions about the tool (fight the urge!). Page one of the help files provides an overview of various basic features such as a simple query of log files, the results of the query being output to an html format, and the generation of a graph representing a query result all via command line instructions.  No point in rehashing these basics here as the documentation does a stellar job.

In my situation, I wanted to do some heavier reporting on the IIS log data and furthermore wanted to put the log data in the context of the application (which conveniently captures user and IP address data to SQL server).  Log Parser installs with SQL Server and MS Access support.

To push query results to SQL Server, here is an example:

LogParser "SELECT * INTO Logs FROM C:\Logs\LogRepository\*.log" 
-iCheckPoint:C:\Logs\LogRepository\LogCheckpoint 
-o:SQL -server:(local) -database:LogWarehouse -driver:"SQL Server" 
-username:dev -password:supersecret -createTable:ON -ignoreIdCols:ON

After installing Log Parser you can run it from Start Menu.  Once the console is open, you can type (or paste) the above code into the command line.  Log parser queries (for the most part) begin with LogParser so be sure to prepend this to your command.  The actual query here is the trivial case in that I’m pushing ALL data into SQL Server.

SELECT * INTO ... FROM ...

More complex and/or specific commands are definitely possible!

The rest of the command is mostly connection-string like configuration data, such as server and database name along with credentials.  The real interesting pieces are -iCheckPoint and the last two arguments -createTable and -ignoreIdCols.

iCheckPoint is a great feature of Log Parser.  When this flag is set a file is created with metadata about the position of the last entry queried.  So if you have 100 log entries and you execute a query for the first time (a checkpoint doesn’t exist) then all 100 records are queried and a checkpoint is created.  A future query will then only execute against new log entries (and files) and the checkpoint is updated. See the documentation for an explanation of how the checkpoint stores data and handles various logging scenarios.

Because I set the -createTable and -ignoreIdCols flags, a table will be created (if it doesn’t exist) and any identity columns found will be ignored.  If the table doesn’t exist Log Parser will create the table with all columns for the entire IIS Log File Format regardless of if the log file contains data for the columns.  This is ideal as you can turn logging options on and off without added maintenance to the table.

LogID is not a convention of LogParser, but instead an identity column I added to the table for my own selfish reasons.  You can easily set LogParser to continuously add logs to your table as they come in, and the iCheckPoint gives LogParser a fighting chance at scalability.  I’ve pushed just over 2 million logs to a SQL table at once and it was a manageable wait – approximately 10 minutes (Core2Quad, 4GB RAM, under existing load).

I can definitely say Log Parser is a great go-to tool, especially when prototyping or scenarios when results are needed/wanted very quickly.  Set-up is effortless, documentation is fantastic, and the learning curve is friendly so don’t hesitate to give it a shot!