With the SharePoint Conference 2012 behind us, I have been reflecting on our SharePoint journey so far…and on the road ahead. And what an incredible journey it has been! SharePoint has allowed AIS to build mission-critical applications for various large federal government agencies and commercial organizations. And not just ECM or document management systems (which are great workloads enabled by SharePoint) but enterprise-class applications for tens of thousands users (such as the FBI’s Delta Project), built using SharePoint platform elements such as workflows, lists, libraries, search, etc.

This blog entry is comprised of two parts. The first part will focus on the SharePoint journey so far. Through a series of short video clips, I will present some of the key insights we have derived over the many years of building custom applications on SharePoint. We will end this the first part with a short demonstration of SharePoint-based Case Management application that brings together many of the key concepts. The second part will focus on the road ahead and the most important enhancements made in SharePoint 2013.

Part One: The Journey So Far…

The SharePoint journey began well before 2007.

It was version 2007 that first staked SharePoint’s claim as an application development platform. Here is the paper we released on January 30, 2007 (the day SharePoint 2007 was launched). We published an updated version of the paper on the day SharePoint 2010 was released.

In order to understand the motivation behind the SharePoint as an app dev idea, it is helpful to take a step back and think about it in the context of composite solution architectures.

(credit for slides used in this video clip goes to an unnamed presenter on the OWA team)

SharePoint: “The Long Tail”

(credit for this slide goes to Arpan Shah, Sr. Director of product management for SharePoint 2010)

One of the canonical usage scenarios for SharePoint App Dev is a “Case Management” application. This video clip discusses common case management requirements and challenges in building such systems.

This next video clip provides a brief discussion of how a case management system may be implemented on top of SharePoint.

Some key features highlighted in the demonstration (coming up next).

Demonstration Part One: Creating a New Case

Demonstration Part Two: Updating a Case

Demonstration Part Three: Multiple Ways of Surfacing SharePoint Data

Hopefully the above demonstration provides some insight into the kinds of applications that can be built using SharePoint platform elements. We can broaden our case management demonstration into a set of generic use cases that we believe should be considered when building applications on the SharePoint 2010 platform:

1. The need to provision more than one website based on a logical grouping (such as department, region or country) rather than have one website that serves all users. For example, a company needs to develop a web application for its partners that allows them access to pertinent sales information. It would be perfectly reasonable to start out with an ASP.NET application; however, as the application usage grows, partners would like to customize the site based on their own unique needs. The partners may want the sales information to surface differently (i.e. grouped by regions vs. grouped by cities), or they may want to co-locate additional applications on the same page (i.e. a tax calculator). Rather than building all of this personalization in code, it is easier to provision a site for each partner that is based on a single common site blueprint. Each partner can then customize their site based on their specific needs via a single code base.

2. The need to manage unstructured content (i.e. documents, web casts, etc.) combined with social networking capabilities. Most modern websites need to manage ever-increasing amounts of digital content. A distinction between the structured content and unstructured content is that the former deals with data that can be viewed and managed using set-based groupings (database views), whereas the latter deals with data that is managed as explicit standalone entities along with the associated metadata. Not only is it important to manage the integrity and security of standalone entities (such as documents); it is also important to manage the relationships among them (for instance, the association relationship between a document and structured application data).

Additionally, provisions need to be made to facilitate business process workflow (routing, approval, regulatory compliance, etc.) and document and folder-level access control and search. The recent trend of social networking makes unstructured content even more challenging. Users are demanding the ability to tag a document, rate it, locate and interact with the subject matter experts related to the document, all in near real-time. Again, rather than building these capabilities into an application, it is far easier to rely on the Content Management Services provided by SharePoint.

3. Organizations seeking to involve the business users and analysts in all aspects of development, not just during the requirements gathering process. One of reasons for the success of SharePoint has been its ability to empower end users. From creating custom views on lists, customizing web parts and building workflows, SharePoint has allowed the end-users to create much of the functionality they need without relying on IT each step of the way. However, in many instances this has led to organic growth that can get out of control easily. By enhancing the collaboration between these roles and providing a model for better IT governance, SharePoint 2010 made further inroads in this area. Consider the following example: it is possible for the business user to customize a live site inside the browser and then save the site as a template. A developer can then import this template into a development tool such as Visual Studio and continue to build on the customization made by the business user. The end product is then available again to the business users.

4. Organizations looking to build departmental applications that are based on external sources of data. While there were a number of approaches for integrating LOB applications within SharePoint Server 2007, a number of challenges remained. SharePoint 2010  BCS made it possible to surface external data sources as SharePoint lists, generate CRUD screens and incorporate external data into the integrated search. External data sources were not limited to relational data but could also include Web Services and plain .Net classes. By incorporating various departmental applications into a single platform, organizations could reduce the total cost of ownership through one common user experience, development and deployment strategy.

Part Two: The Road Ahead 

This brings us to part two of this blog.  As you may have read by now, Microsoft has announced a slew of new features for SharePoint 2013 that is sure to further enhance SharePoint’s credentials as an application development platform.  In this section of the blog post we will look at some of the key enhancements that we believe are going to be very important for folks building custom application using SharePoint 2013.

1. Search-driven applications. While lists, sites, site collections etc. have served as useful logical containers when building custom SharePoint applications, application developers have had to deal with the challenging task  of aggregating data across these containers. For example, in our demonstration earlier, in order to aggregate tasks across site collections (each case is separate site collection), we developed a custom service application that in turn relied on the change log. Search is much better suited for solving this problem since all of the SharePoint content is available to search.

However, up until now, search was not seen as a viable alternative due to potential staleness in data due to the “lag” between the crawls and limited query capabilities associated with search. All this is changed in SharePoint 2013 with the ability to setup continuous crawls, execute rich search based queries and new ways to tag content.  As a result of these changes we see search being leveraged as an important building block when building custom SharePoint applications.

2. App Model. Perhaps the single biggest architectural change in SharePoint 2013 is the ability is host custom pieces of logic such as event receivers and timer jobs outside of SharePoint’s execution context. The motivation for this change should be obvious. The SharePoint team wants you to host customizations outside of the SharePoint process space, thus allowing the SharePoint farm to be not only easier to manage but also easier to upgrade in the future. In order to make it possible to host customizations outside of SharePoint, Microsoft had to ensure that SharePoint functions were accessible remotely. So unlike the previous versions where only a limited subset of the capability was accessible remotely via web services, SharePoint 2013 makes majority of the SharePoint functionality including managed metadata, search, workflow available to remote consumers.

Pivoting on this important architectural change is the notion of the app model within SharePoint. Think of a SharePoint app as a self-contained unit of functionality that can be discovered, purchased and installed from a public (Office.com) or private store.  Based on where the custom code is hosted, SharePoint apps can be classified into three distinct styles:

i) SharePoint-hosted apps can only include JavaScript based custom code that gets executed on the client side.

ii) Auto-hosted apps rely on custom code such as workflows and timers that execute within the Windows Azure platform code.

iii) Provider-hosted apps rely on custom code that executes on dedicated server(s) outside of the SharePoint farm. At first, the SharePoint app model may appear limiting when compared to the farm-based solutions we have all relied on heavily on, up until now.

However, by rethinking and adjusting some of the common patterns, you will find that most customizations can be accomplished using  SharePoint apps. Consider the well-known pattern for implementing a periodic task within SharePoint. We have relied on timer jobs for realizing this pattern.  In the app-based model, this is something that can be easily (and more robustly) implemented in Windows Azure.

Finally, all parts that make an app are contained within the app itself, installed and uninstalled as a unit. So when a user purchases and installs an app, there is no expectation that one or more prerequisite pieces of software have already been installed on the SharePoint farm. Likewise, when a user uninstalls an app, all parts that make up the app are removed.

3. Workflow as its own subsystem. Workflow is perhaps the most commonly used feature when building applications for SharePoint (ok… may be after the lists and libraries). However, the fact that workflow instances are typically executed within the w3wp process (they can also be queued to run inside a separate worker timer process) has been limiting in terms of the number of concurrent instances, number of events processed, etc. While there are a number  of knobs one can turn in order to calibrate the throttling and batch size, the real solution is to host the workflows outside of SharePoint.

That is exactly the case with SharePoint 2013. Workflows are hosted within a Workflow Manager client outside of SharePoint. Additionally with the move to WF 4.0, we believe that  workflows will be much more robust and with the emphasis on declarative workflows, much easier to author.

4. Support for mobile devices. Success of any development platform hinges on its ability to support an ever growing list of devices. SharePoint 2013 does just that. With the optimized HTML5-based mobile browser experience, ability to associate multiple master pages to a single site ( in order to support multiple form factors), support for push notification and the addition of geolocation list datatype to make it easier to implement location awareness, SharePoint 2013 makes it easier to support custom applications that target a wide array of devices.

5. Revamped Access Services. You are probably surprised by this item. With so many other worthy enhancements to choose from (enhanced API , performance improvements, BI enhancements), why do we choose to highlight the enhancements to access services?  As stated before, we consider the ability to build departmental applications that rely on external sources of data as an important use case for SharePoint as an application platform. Unlike Access Services in SharePoint 2010 where data resided in SharePoint lists, SharePoint 2013 provisions a new application database that stores the data, view, queries and macros.  In fact, the application database can be created on a separate SQL Server 2012 server.

As you can imagine, this provides ultimate flexibility for managing and scaling Access applications in SharePoint 2013. Imagine taking an Access 2010 database and publishing it to SharePoint 2013.  Simply by virtue of the publication process, an Access database has been transformed into a scalable server application!