Web Applications in the age of Azure

Increasingly, I’ve found myself turning to Microsoft’s Azure platform for new web projects. The convenience of on-demand infrastructure, combined with a usage based payment model is compelling; but what makes Azure shine are the rapidly expanding set of supporting services available.

When you fist start writing Azure WebSites, you quickly run into complexities that are often unfamiliar to developers who only ever targeted on-prem IIS servers. You can’t rely on the local file-system, sites span multiple server instances, you have to ship dependencies along with your application’s code, and so on.

These issues aren’t new for those who develop for web-farm environments, but even the smallest Azure WebSite forces you to consider complexities of scale –issues that plagued only the largest of projects in the past.

Fortunately, Azure provides a complete set of services to address those complexities, and using them is fairly easy. Caching services handle shared session state between instances of your application. Azure storage substitutes for the local file system. Azure WebJobs handle background processing and scheduled tasks.

There are also services analogous to most of the common external dependencies; Azure Search, Azure DocumentDb, Azure SQL, Azure AD, etc. Some of these are so similar to their on-prem equivalents that using them is largely transparent, but most act more like a ultra-modern replacements for their older, on-prem cousins.

If you need a service Azure doesn’t offer directly, you can always spin up an Azure VM. The Azure Gallery has pre-built VMs for tons of common services, or you can roll out a standard Linux or Windows VM and install whatever third party software you need.

It took me a little time to learn my way around the Azure platform. At first, I resented the additional effort, especially in small projects. After the rather modest initial learning curve though, I have found that Azure provides far more than simple replacements for the same old services I’ve always used. They offer a lot of additional value that I’ve never had easy access to before.

Here is an example. The first time one of my Azure applications needed to handle user file uploads, I found it cumbersome to use Azure Storage. I had to provision a storage account, pull down the nuget packages, then configure my application to talk to it. It only took about 20 minutes the first time, but it seemed like a hassle compared to just calling into System.IO.

After getting it all hooked up though, I discovered that Azure storage wasn’t difficult to use, and it eliminated a lot of common problems. I no longer had to worry about mapping server paths, dealing with file and folder permissions, or file contention issues. It just works, and it scales without thought.

Still later, I came across the need to write information to a queue so a background process could act on it later. Normally I’d have to create a table in my database, or drop custom files on the file system to share information with the background process –or worse, deal with Microsoft Message Queues (I still have nightmares). After having setup Azure storage for the file uploads though, I also had access to Azure Storage Queues at my finger tips.

I had similar results when I had to setup a cache service for handling session state. It was a tad inconvenient, but when I needed to cache data I’d fetched from the SQL database, I already had a super-easy, super-fast, cloud-scale caching service right there waiting for me.

Sure, I can setup local servers and services that can do any of these things, but in Azure I don’t have to. These services are already there, all I have to do is turn them on.

It has gotten to the point where I really miss Azure whenever I’m working with on-prem applications. I wish IIS could run WebJobs, and that my local ADFS server supported OpenID Connect. I want a search server on-prem that doesn’t require voodoo-devil-magic to setup and maintain. Working outside of Azure has become an inconvenience.

TicketDesk 2.5 – Progress and Developer Notes

I wanted to give everyone an update on the progress of TicketDesk 2.5, and take a few minutes to explain the general architecture that’s starting to take shape over on the develop branch at CodePlex.

Projects and Architecture:

The Database supports any version of SQL Server 2008 or higher, including Azure SQL (but not compact edition). The design of TicketDesk would also fit a document database very well too, so TD may eventually end up having a RavenDB and/or Azure DocumentDB variant.

The Schema is managed by code-first migrations, with the execution of migrations handled by startup initializers and/or on-screen admin tools. It uses multi-tenant, code-first migrations. The identity and the business domains have separate contexts and models, each of which are tenants within the same DB. Similarly, search and email will be separate tenants too.

TicketDesk.Domain is the core business layer. It has no dependencies on web-specific frameworks, azure frameworks, or specific security providers. It does have a dependency on entity framework, and leverages EF as a full scale business service platform, not just as a data access technology.

I call this a “pervasive EF Domain model”. It’s a lot like most domain models you see described in any .net book or tutorial, but without unnecessary abstractions to hide the entity framework components.

The Entities directly contain their own business logic, while DbSets act as generic repositories. Custom repository functions are provided mostly through extension methods. Some extensions are defined in the domain assembly, while those with web specific dependencies are defined within the web application instead –which is why a DI framework or IoC container has’t been necessary within the domain project.

The DbContext is treated as the root business service, and it provides the unit-of-work pattern. Truly cross-cutting business logic will be handled directly by the DbContext, or through extension methods and helper services.

The application doesn’t need a formal DDD style design, but with search and email breaking out into separate services, the design is headed in that general direction. By the time TD 3 is complete, there will likely be a formal service bus mechanism handling communications between separate root business contexts, and this will drive a DDD style eventual consistency pattern.

The first parts of that design will appear in 2.5 with the queue mechanisms that support email and search. It likely will never adhere to the full DDD battery of principals and patterns, but it will borrow heavily from those designs where they make sense.

TicketDesk.Domain.Legacy exists only for conversion of TD2.1 databases… it’s just an isolated container for EF migrations and migration helpers. It has a completely custom initial migration that upgrades a TD 2.1x database to the starting TD 2.5 schema. After that, the regular TD 2.5 migrations from the main domain assembly will bring the database up to the final schema version.

TicketDesk.Search encapsulates all search related functionality. Currently it has local lucene and azure search providers. This could be decomposed into separate assemblies, but I haven’t seen the need yet to go that far with it yet.

Currently, it also fakes a “queue” based mechanism, but before TD 2.5 ships this will be replaced by a more formal queue management system.

TicketDesk.Identity encapsulates all non-web/owin specific security functionality. This is an odd one architecturally, since much of the overall identity system does need dependencies on owin middleware components. Right now, the core EF and aspnet.identity stuff is isolated here, but the web app layers on additional user and role managers.

Much of this is boilerplate identity code borrowed from stock samples, so as I refactor it for TD’s specific needs, I’m thinking seriously about moving all of the identity stuff to this assembly, and letting it have owin dependencies. I’m waiting until I tackle ADAL and/or OpenID Connect (for AD federated security) before I decide for sure, but I don’t want to go as far as to write a custom abstraction layer for this thing.

Either way, the identity stuff will remain decoupled from the business domain assembly entirely.

TicketDesk.Web.Client is the main web application. Nothing special about this design, except that it is an ultra-modern use of the latest MVC stack. It uses Owin/Katana, Aspnet.Idneity, and all the new toys like that.

It makes light use of the Dependency Resolver, with Simple Injector as the underlying DI service. The UI is written to bootstrap 3, with some light jQuery and custom javascript here and there. It does use the jquery unobtrusive ajax stuff for partial view rendering –a concession that makes it easier to port the old UI without having to completely re-invent the entire design.

Mail / WebJob: I haven’t implemented these yet, but email will be split out like search, and will use a DB queue table when running on-prem, and will use Azure Queue Storage when running on Azure. On Azure, mail delivery will be handled by Azure WebJobs.

What I’m working though now, or in the very near future:

Building out the rest of the core UI:

I’m working my way through porting the UI to MVC 5, Razor, and bootstrap 3 now. TicketCenter is close to done, and I’m currently working on the New Tickets screen. Afterwards I’ll get into the main ticket viewer/editor.

The goal is to reproduce most of the current behavior exactly as it exists in TD 2.1. Later, I’ll revisit the behavior to further streamline it. The Viewer/Editor is an command/action design, rather than a view/edit/submit design, so it lends itself better to an MVC style design pattern anyway. There are a few activities that do need some smoothing out, but most of that will wait until TD3.

Main Text Editor:

TicketDesk has alternately used an HTML WYSIWYG editor, or a Markdown editor in past versions. I’ve always intended to support both at the same time, but for some reason never seem to get around to it.

TD 2.5 will offer both options though. I have the PageDown editor (markdown) working in the new ticket editor already, but I also plan to implement a limited html editor –probably SummerNote. The back-end has always supported both kinds of content, so all I have to do there port the current code over from TD 2.1.

There will be admin settings where the admin can choose which editors are enabled, and which is the default. If multiple are available from the settings, each user will be able to set their own preference.

Ticket Center Lists Editor:

TD 2.1’s ticket center is designed around the idea of user tailored lists (my tickets, history, open tickets, etc.). In TD 1, you could customize these lists, and I intended to make that a feature of TD 2 as well. A customization screen appropriate for non-technical users is a big challenge, and TD 2 also had a significantly more complex way of managing settings for these lists, so I never had time to make that happen the way I wanted.

For TD 2.5, we will at least have tools to let admins change the default list definitions, maybe by just letting them edit the raw JSON if nothing else. A full end-user custom list designer would be nice, but it will probably have to wait on TD3.

Images & Attachments:

TicketDesk has traditionally only supported file attachments, but not embedded images via the text editor. It stores those files in the database directly. This makes it very easy to move the database from one server to another, and also simplifies backups as well. While performance hasn’t been a problem, this isn’t the best approach in cloud or hosted deployments… if nothing else, the cost of renting SQL storage space is very high compared to file or blob storage.

TD 2.5 will need a plugable storage provider that can store attachments on the local file system (or file share ), or in Azure Storage Blobs. The legacy database migration feature will need to move existing file data to the storage provider before it removes the old attachments table.

Eventually I hope to have support for popular file storage solutions like dropbox and friends.

The front-end text editors will ideally be able to handle inline image uploads, though I don’t think a full media gallery is really necessary –image re-use almost never happens with help desk type systems anyway. If an uploaded image is bigger than a certain size, it should be swapped for a thumbnail in the editor text, and the full image included as an attachment instead.

Eliminate MVC Areas:

I don’t like what areas do to MVC’s routes and navigation, especially with the newer attribute routing stuff. It was cool back in the days of MVC 2 and portable areas, but has outlived its usefulness in MVC 5.

Right now, I do have a couple of areas setup in the code, but I plan to remove these very soon, then cleanup the action and url links so they don’t have to specify an area as custom routedata each time.

Admin:

TD 2.5 will have a “first-run” setup experience. There is a sort of stub for this already, but this will be replaced by an expanded wizard to walk the admin through setting up everything. Most important is setting up the database, security, email, and search features.

Part of this experience will include directing the admin on how to make a few changes in web.config that can’t be automated easily, like adding custom machine keys and such.

Localization (pre-support):

I’ve already have this worked out in TD3. The UI text will be supplied from resource files, instead of being hard-coded. I haven’t quite gotten into the string-heavy part of the system yet, but as soon as I do, I’ll start moving all of the text to resource files. TD 2.5 itself will not provide a full internationalization experience out of the box, but I want to go ahead and get the code in shape for the features coming in TD3. This will also make it super easy for developers to localize the system themselves.

Thanks to everyone for their support and patience!

Dev Diary – Scaling search for an Azure WebSite

azuresearchAdding real search capabilities to a custom web application is never easy. Search is a complex and deeply specialized area of development, and the tools available to us regular developers are monstrously complex.

With TicketDesk 2, I used the popular Lucene.net library to provide search. Ported from Apace Lucene (Java), this is the core technology that powers almost every popular search service, appliance, and search library on the market.

Once you’ve tackled the initial learning curve, Lucene.net isn’t all that difficult to leverage in a simple system like TicketDesk. It is freakishly fast, super flexible, and is a powerful search solution –not quite Google good, but close enough for most applications.

The problem with Lucene is that the design revolves around indexes stored on a traditional file system. There are 3rd party extensions that let you store the indexes in a database or in the cloud, but internally these all mimic the behaviors of a file system –that’s just how Lucene works.

You can have many components querying an index at the same time, but only one can write to an index at a time. Normally this single-writer limitation isn’t a huge problem. You code your application so it creates just one writer instance, then share it with any components that want to make an index update. As long as you keep things synchronous, it tends to work fine.

And here lies the problem. TicketDesk 2.5 and 3.0 are designed to run at scale, and will ship ready for deployment to the cloud as an Azure WebSite. In this scenario, there can be several instances of the application running at the same time, each needing to write to a single, shared Lucene index.

I spent a full week trying to find a way around the single-writer problem. WebSites in Azure shouldn’t write to their filesystems. Anything written locally is volatile, and vanishes whenever Azure automatically moves the site to a different host. So, I started with the AzureDirectory library for Lucene, which lets you store the search index in Azure blob storage. This works well, and gives Lucene a stable place to store shared indexes in the cloud.

The second problem was keeping multiple web site instances from writing to the index at the same time. Even though the index is in blob storage, Lucene still demands an exclusive write lock. Each websites can see when the index is locked by another writer, but there isn’t a way to know if the lock is legitimate, or an orphaned lock left behind when some other instance went down unexpectedly.

The only easy solution is to make sure there is a separate application to handle all index writes, and that there is only a single instance of that application running. You can scale the websites or other clients, just don’t scale the index writer application.

WebJobs were designed specifically for handling background on behalf of Azure WebSites, so I started there. Each website would queue index updates to an Azure Storage Queue, then the WebJob could come along and process the queue in the background. But WebJobs scale with the websites, so if you have multiple websites, you also have multiple webjobs. Hopefully, in the future MS will give us the ability to scale webjobs independent of the websites they service.

So the only remaining solution would be an old fashioned worker role. They scale independently –or in this case, can be instructed not to scale. This works well, but I just don’t like the solution. Effectively, the worker role ends up being a half-ass, custom search server. It costs a decent amount of money to run a separate worker role instance, plus it complicates the deployment and management of the entire application.

Failing to find a way to continue in Azure with custom Lucene indexes without a centralized search server, I figured I’d just design TicketDesk to take advantage of the existing Azure native solution –Azure Search Services. It is easy to code against (relatively), and there is a free tier that should be suitable for most smaller shops. For larger shops, the costs of a paid Azure Search tier is still reasonable when compared to the costs of a dedicated worker role.

So, out of the box, TicketDesk 2.5 will include at least two search providers; a Lucene provider for on-premise single instance setups, and native Azure Search for cloud deployments. I will eventually add an alternative for on-premise webfarms, and non-azure cloud VMs. In the meantime though, you could still scale in your own data-center by using Azure Search remotely, or stick with Lucene and manually disable the search writer on all but one instance of the site in the webfarm.

One additional note of interest: Azure Search is still in preview, and it doesn’t have an official client library for .Net yet. There are two 3rd party client libraries though; Reddog.Search and Azure Search Client Library. Both are free as NuGet pacakges, but only Reddog.Search has a public open source repository. Also, Reddog has a management portal you can run locally, or install as an Azure WebSite extension.

 

Entity Framework – Storing a complex entity as JSON in a single DB column

jsonDuring the development of TicketDesk 2.5, I came across an unusual case. I wanted to store a large chunk of my entity model as a simple JSON string, and put it into a single column in the database.

Here’s the setup:

I have an entity that encapsulates all of a user’s display preferences and similar settings. One of those settings is a complex set of objects that represents the user’s custom settings for list view UI pages. There can be many lists, each with separate settings. Some of the settings for a list contain collections of other objects, resulting in a hierarchy of settings that goes three levels deep

I didn’t want to represent these settings as a relational data model in the database though. Using EF’s standard persistence mapping conventions, this collection of settings ends up being spread across six  tables. The TSQL queries to access that data would be rather slow and cumbersome, and the relational model doesn’t add any value at all.

Instead, I just wanted to serialize out the entire collection of settings as a single JSON string, and store it in one column in the user settings table. At the same time though, I wanted the code behave as if this were just a natural part of my regular EF entity model.

The solution:

The solution was to use a complex type, with some fluent model binding magic, to flatten the hierarchy into a single column. The heirarchy itself is represented as a custom collection, with a bit of manual JSON serialization/deserialization built-in.

I got a pointer the right general direction from this SO post, which saved me a bunch of time when approaching this more advanced scenario.

First, let’s take a look at the root entity here:

public class UserSetting
 {
 	[Key]
    public string UserId { get; set; }

    public virtual UserTicketListSettingsCollection ListSettings { get; set; }
}

This is the only entity which will map to its own table in the DB. The ListSettings collection is the property I want persisted as JSON in a single column.

Here is the custom collection that will be stored:

public class UserTicketListSettingsCollection: Collection<UserTicketListSetting>
{
    public void Add(ICollection<UserTicketListSetting> settings)
    {
        foreach (var listSetting in settings)
        {
            this.Add(listSetting);
        }
    }

    [JsonIgnore]
    public string Serialized
    {
        get { return Newtonsoft.Json.JsonConvert.SerializeObject(this); }
        set
        {
            if (string.IsNullOrEmpty(value))
            {
                return;
            }

            var jData = Newtonsoft.Json.JsonConvert.DeserializeObject<List<UserTicketListSetting>>(value);
            this.Items.Clear();
            this.Add(jData);
            
        }
    }
}

This is a collection type, and is inheriting generic Collection<T>. In this case, T is the UserTicketListSetting type — which is a standard POCO wrapping up all of the settings for all of the various list views in one place.

Some of the properties inside UserTicketListSetting contain collections of other POCOs. The specific details of what’s going inside those classes doesn’t matter to this discussion, just understand that it results in a hierarchy of related objects. None of the properties in that hierarchy are marked up with EF attributes or anything.

The only magic here is that we have a Serialized property, which manually handles converting from/to JSON. This is the only property that we want persisted to the database.

To make that persistence happen, we will make UserTicketListSettingsCollection an EF complex type, though not by using the [ComplexType] attribute. Instead, we’ll manually register this complex type via the fluent model builder API.

In the DB Context this looks like this:

protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
	modelBuilder.ComplexType<UserTicketListSettingsCollection>()
        .Property(p => p.Serialized)
        .HasColumnName("ListSettingsJson");
}

This just tells EF that UserTicketListSettingsCollection is a complex type, and the only property we care about is the Serialized property. If there were other properties in UserTicketListSettingsCollection, you would need to exclude them with something like:

modelBuilder.ComplexType<UserTicketListSettingsCollection>().Ignore(p => p.PropertyToIgnore);

And that’s all I needed to get EF to store this entire hierarchy as a single JSON column.

Using this model in code is just like using any other EF entity. I can query it with LINQ expressions, and SaveChanges on the DbContext updates the JSON data in DB just like any other entity. Even the generation of code-based migrations works as expected.

It took a LOT of experimentation and digging to figure out how to make this work, but the implementation is rather simple once you know how to approach the problem.

This also reflects the amazing power and flexibility of Entity Framework. EF can be extended to fit very advanced scenarios even when the designers didn’t anticipate them directly.

You can see the full implementation of this in TicketDesk 2.5. Currently, TD 2.5 is in alpha, so look to the develop branch in source control on CodePlex. You will find this example, as well as several variations in the TicketDesk.Domain assembly.

TicketDesk 2.5 – Coming soon!

TicketDesk-2.5While I’ve been working on TicketDesk 3, the code for TicketDesk 2 hasn’t been getting any younger. Since TD3 is still a ways out from a production release, I’ve decided to release a major overhaul of TicketDesk 2 in the meantime. The new TicketDesk 2.5 release will bring the technology stack of TD2 up to the latest version of Microsoft’s Web Platform. Several of the changes will be derived from code developed for TD3.

I am targeting the end of October for a beta version, with a final release in mid to late November (subject to change, as always).

Here are the major changes I have planned so far:

Remove AD Security:

This release will not support direct integration with Active Directory. This was a popular feature in TicketDesk 1 and 2, but it has also been a major problem area as well. Instead, TD 2.5 will only support AD indirectly, via federation (ADFS for example) or through integration with an external identity server (like Azure AD).

Modernized Local Security:

TicketDesk 2.1x still uses the ancient SqlMembership providers that shipped with .Net 2.0 back in 2005. Authorization and identity have come a very long way since then, so TD 2.5 will be upgraded to the newest Aspnet.Identity framework version. It will also provide on-screen management tools to help administrators migrate existing user accounts from their legacy TD 2.1x databases.

UI Changes:

TD2.1x was built on Asp.net MVC 2, with the original Asp.net view engine. This isn’t very well supported by recent versions of Visual Studio, and most developers have long since abandoned the old engine in favor of Razor. I don’t plan many major changes to the general UI’s behavior or appearance, but by re-implementing on Razor I can bring the project into compatibility with Visual Studio 2013 and current Asp.net coding standards.

There will be some minor changes to the UI. I will be removing some of the crusty old JQuery components, and updating the styles to take advantage of newer CSS features that weren’t widely supported when TD2 was first built.

Entity Framework Code-First:

TicketDesk 2.5 will move from Entity Framework 4 database-first models to Entity Framework 6 with a Code-First model. EF Migrations will provide ongoing schema management from here out, including the ability to migrate legacy TicketDesk 2.1x databases. Along with this change, TD 2.5 will also include on-screen database management utilities for migrating legacy databases, seeding demo/test data, and so on.

This refactoring will also bring TD 2.5 in line with the technologies backing TD 3, which will greatly simplify future upgrades.

Eliminate MEF:

The managed extensibility framework has continued to evolve, but it still isn’t a very good choice for Dependency Injection concerns in a web application. Instead, TD 2.5 will use Simple Injector for IoC. Some of the simplifications in the back-end should also reduce the reliance on dependency injection techniques quite a lot.

Improved Email Notifications:

Several improvements to Email Notifications are planned. Most of these are intended to give administrators greater control over how and when TicketDesk sends email notifications. This will include better on-screen testing and diagnostic tools for troubleshooting email related issues.

Multiple Projects:

TicketDesk 2.5 will support multiple projects, which let you handle tickets for different operations, projects, or products in isolation. You will be able to move tickets from one project to another, and can turn off multiple project support  entirely if you don’t need the functionality. I do not know yet if I’ll support user permissions on a per-project basis in this version, but TD3 will certainly provide that functionality.

Watch/Follow Tickets:

Users will be able to watch or follow tickets without having to be the ticket’s owner or assigned user. This will allow these users to receive notifications of changes as the ticket progresses.

Azure Deployments:

TicketDesk 2.5 will be deployable as an Azure WebSite. Currently, there are several issues that make deploying to cloud or farm environments tricky. The biggest are that the Lucene search indexes are stored on the file system, and the old SqlMembership providers are not compatible with the features provided by Azure SQL. These issues are not insurmountable for experienced .Net developers, but deployment to web farms or cloud providers is not currently an out-of-box capability of the system.

To make TicketDesk play well with cloud environments, a pluggable storage provider will be used for any features needing access to non-database storage. When deployed to single instance environments, TD 2.5 will use the file system, but you will be able to reconfigure it for Azure Blob storage when deploying to the cloud. Attachment storage will be moved out of the SQL database to the new storage provider as well.

The only hold-up for Azure SQL is the membership system, but the newer Aspnet.Identity framework fully supports EF migrations and is compatible with any EF data provider –including Azure SQL.

Early pre-alpha code is already committed to the CodePlex repository, and will be updated regularly as work continues. Right now, there isn’t much to see. I’m still working on the basic plumbing for database and identity management, so there are no TD specific user facing features yet. As soon as the back-end is shored up, I’ll start porting in TD specific UI features.

A demo version of the site will be available soon, hosted on Azure. I just have to workout a few minor details related to resetting the database and providing sample seed data, then I can open the demo to the public.

TicketDesk 3 Dev Diary – Angular it shall be!

which wayIt has been a while since I updated everyone on TicketDesk 3. I took a break to wrap up several other projects and to search for a new day job, but now I’m back to working on TicketDesk 3 again.

I haven’t been idle though. I spent much of this spring working with Asp.net identity, OWIN authentication middleware, and all the federated identity goodness — Azure AD, Azure Graph API, ADFS, Azure AD Sync, WsFederation, oAuth, OpenID Connect, etc.

I now have a decent grasp of what’s going on in the world of modern authentication and identity. There is a LOT happening in this space right now, and the Asp.net stack is right there on the bleeding edge of it all.

Unfortunately, documentation and guidance on how all the new security pieces fit together in real world apps is sparse, but I’ve learned enough now to be comfortable that I can get TD3 to handle multiple authentication scenarios.

Over the last several weeks, I’ve also been working deeply with Angular.js. I’ve dabbled in SPA’s in the past a bit, mostly on the Durandal, knockout.js, and breeze stack –which the current TD3 Alpha uses. This space has also been moving fast, and Angular in particular has been gaining traction like crazy.

For me, the impetus for examining Angular in more depth was the announcement that Durandal’s principal author had joined the Angular team, and that Durandal will merge with the next version of Angular (2.0). I really liked Durandal, but Angular is what all the cool kids are using –it just makes more sense to go with the tide, rather than stick with a soon-to-be-obsolete framework

After working with Angular.js a bit, I decided to go ahead and move TD3 over now, before the UI gets any larger or more complex.

So far, I’m enjoying Angular. It is a more complex platform than Durandal, so getting my head around it required significant re-training. But I’m finding that it is an amazingly productive platform.

As before, I’m basing the early TD3 platform on the courses from John Papa, and his Hot Towel Angular packages. Hot Towel made a good starting point for Durandal, and it makes an even better one for Angular. This time though, I’m not sticking as close to the Hot Towel provided UI bits. Instead, I’m working with a much more advanced theme from the wrap-bootstrap project.

Here’s the short-term plan:

  • Start with the Hot Towel Angular packages.
  • Mix in a gutted version of the new theme –leaving just the parts I intend to use.
  • Hook up Asp.Net text resources to i18Next and Angular’s internationalization filters.
  • Integrate client bearer token security with asp.net identity and OWIN middleware.
  • Setup sever-side Breeze and the Breeze.js library to move the data around.
  • Put the SignalR stuff back in.
  • Tackle production js/css minification
  • Clean it up, and package it as a starter kit
  • Build out the rest of TD3 on the new platform.

I have a couple of upcoming projects with platform needs, so this base-line starter kit will serve as a common ancestor . If it’s good enough, I might even make it a public nuget package or something.

I’ll commit code to github and codeplex as soon as I have the platform in a usable state, and can afford to buy the extended license I need to redistribute the theme I’m using.

 

TicketDesk 3 Dev Diary – Update: One AspNet and Aspnet.Identity

Since most of what I’m working on now is being done in my private repository, I wanted to give everyone a quick update on TicketDesk 3′ s progress.

I’m still working on TD3, but late in the fall it became apparent that I needed to wait on the RTM version of Visual Studio 2013 and the new Asp.net Web API 2 and Identity bits. Now that this stuff has all been released, and most of the dependencies have caught up, I have resumed work on TD3.

The main challenge is incorporating the new Aspnet.Identity framework. I am thrilled that Microsoft finally replaced the old security providers, but the transition to Aspnet.Identity is not all roses. The new framework is not as well documented as I’d like, and guidance on advanced uses is still thin. It is also fairly complex framework that requires a good bit of manual coding, especially when used in conjunction with a SPA front-end. Fortunately for me, Yago Pérez Vázquez has created a template project called DurandalAuth that does exactly what I’ve been trying to do with TicketDesk…. combine the bleeding edge versions of Aspnet.Identity and Web API 2 with the bleeding edge versions of Durandal, Breeze, and Bootstrap.

In fact, his template is so good, that I’m pretty much building TicketDesk 3 on top of his template instead of just porting the authentication stuff into my existing TD3 project… and this is why the project hasn’t been pushed into the public repository just yet… it’s a new project that doesn’t yet have all the features from the old one.

There are a few things about the DurandalAuth template that I’m not so sure about; the use of StructureMap instead of Ninject for IoC, and fact that he’s layered the back end with a custom repository and unit of work pattern… but overall, the design is generally the same as what I had been designing for TD3; except that he’d also implemented the new asp.net identity bits. The template also includes some SEO stuff that isn’t relevant to TicketDesk 3, though I may leave it there in case people want to make use of it.

At present, I’m in the process of combining the new project with the code I’ve already written for TD3, and adapting the design to TD3’s particular needs (internationalization for example). This will take a few more weeks, but once I’m done I will be able to push the new project to the public GitHub repository for everyone else to look at.

The technology stack for TD3 is now complete; and includes the following:

  • Asp.net Web API 2
  • Durandal 2 & Breeze
  • i18Next
  • Aspnet.Identity
  • Bootstrap 3
  • SignalR 2

The only major design element that I’ve yet to work out completely is using ACS and/or ADFS security. The identity implementation for Web API 2 uses a different mechanism (organization accounts) for integrating with ACS and ADFS; so I’ll have to find a way to smash the two options together; and provide enough internal abstractions to where either configuration is possible with minimal additional code.

TicketDesk 3 Dev Diary – Thoughts on Security

hunts point, the bronx, new yo... by andre dos santosIn this post, I’m brainstorming new approaches to handling security in TicketDesk 3, and reviewing the limitations of TicketDesk 2’s old approach.

Security in  TicketDesk 2:

TicketDesk 2 shipped with support for either active directory, or local security. Local security via the built-in membership providers worked reasonably well for most deployments, but the AD option has always been problematic.

Windows authentication, in conjunction with the WindowsTokenRoleProvider, works well enough for the basic question, “who are you?”, but windows authentication severely limits in the amount of information you can obtain about the user. You get the windows account name and a list of the user’s AD groups –and that’s pretty much all you get.

Some features in TD required more than that. It needed display names and email addresses, and it needed to list all users in some AD groups; information the built-in providers are incapable of obtaining.

To support these requirements, TicketDesk 2 used a complicated background processes for AD concerns. It maintains a local cache of AD data to reduce the number of queries, and moved AD query processing off of the user servicing request threads.

Even with the abstractions in System.DirectoryServices, querying AD is painfully obtuse. In addition, response times for AD queries are often measured in the tens of seconds –even in single domain environments. You can’t reasonably query AD in real-time without bringing the entire application to a standstill. The custom module alleviates these issues, but was a  complete nightmare to code and maintain.

The other significant problem has been the difficulty of extending the old system to support more features, such as custom roles and permissions. This is why TicketDesk still uses hard-coded roles with pre-defined permissions –I simply never had the time or energy to re-factor the AD side of the system.

Security technologies of the present:

I’ve played a bit with the SimpleMembership and Universal Providers, which have replaced the original SqlMembership providers. ASP.NET’s security system is still based on the same provider framework that shipped with .Net 2.0 back in 2005. Sadly, the framework is a fundamentally flawed system, and in need of a complete redesign.

In recent years, there have been two attempts to modernize the ASP.NET providers, but neither really addresses the underlying problems of the core framework itself. Both are just extensions layered on top of the original infrastructure.

Universal Providers are an evolutionary upgrade of the original SQL providers from ASP.NET 2.0. They mainly just move data access into Entity Framework, which allows them to work with any database back-end supported by EF. Otherwise, there isn’t a much of a difference between them.

SimpleMembership was created for ASP.NET WebPages, but has become the officially recommended replacement in MVC projects as well. This is the default security provider in the newer MVC project templates that come with Visual Studio 2012. As I wrote in another post though, this system has some deep design flaws of its own. You almost have to learn how to hack up a customized version of SimpleMembership just to make it useful in any real-world scenario.

The good news is that both SimpleMembership and Universal Providers include oAuth support backed by the popular DotNetOpenAuth libraries. So, while flawed and build on a shaky foundation, at least they have been modernized.

Traditional windows authentication itself has not changed in any significant way, and there have been no official replacements for the windows security providers that I know of. ASP.NET still uses the same windows provider options that shipped in .Net 2.0, all of which are spectacularly terrible.

The major action for Active Directory authentication has been with Active Directory Federation Services. This is an STS token server that exposes AD to clients that support WS-federation. The Windows Identity Foundation (WIF), originally introduced in WCF, was pulled directly into the core of the .Net framework with the 4.5 release. This provides the infrastructure ASP.NET needs to leverage federated authentication and authorization. So, if you setup federation in ASP.NET you can easily connect it to ADFS to authenticate your AD users. Best of all, ADFS can share any information stored in AD.

In Windows Azure land, you have the Access Control Service (ACS), which is also a WS-Federation token service. From ASP.NET, connecting to ACS is similar to connecting to ADFS. The real advantage of ACS is that it can act as an aggregator and proxy for any number of other external authentication sources. For example, you can connect your ACS to your domain’s ADFS service and to oAuth providers.

The new identity features in .NET 4.5 also incorporate claims based identity as a pervasive and normative feature used throughout the entire framework. You don’t have to use WS-federation in order to use claims based identities, since all identities are now claims based.

Ideas for TicketDesk 3:

For TicketDesk 3, I hope to support the following authentication options:

  • local user accounts (forms with user registration)
  • oAuth (sign-in with google, facebook, twitter, etc.)
  • Windows (AD or local system accounts)
  • WS-Federation (ADFS, Azure ACS, etc.)

My challenge is to find a way to bring all these pieces together in TicketDesk 3 in a sane way.

I want TicketDesk’s business logic to make use of claims based identity for its authorization decisions. Since claims are baked in, this seems to be the most sensible way of doing things. The key will be to map all users, regardless of how they are authenticated, to a common set of claims that TicketDesk understands. The business logic can then rely on having those claims attached to every user’s identity.

For authentication, I have little choice but to rely on existing mechanisms. I am not a crypto expert, and have no intention of rolling my own password and user verification code. But, to support multiple authentication sources, I will likely need to roll my own module to intercept user requests, and dynamically route them to the appropriate pre-built authentication mechanism as needed at runtime.

Each of the possible authentication systems will produce a ClaimsIndentity, but the structure and content of those claims varies. Some authentication generates very few usable claims, while others may contain all the claims TicketDesk could ever need. To bridge the gap, TicketDesk will need an identity transformer. This component will map incoming identities, obtained during authentication, to a TicketDesk identity with all the claims TicketDesk needs. Since some claims cannot be obtained during authentication, TicketDesk will need to obtain any missing data from the user, or an administrator, then store and manage that data locally.

As you can tell, this is all in the early conceptual stages. I’m doing a lot of experiments and research, but I’m at the point where I have a good grasp of most of the pieces that I’ll need to smash together.

I have several other projects with similar security requirements, and I’ve grown tired of approaching each new project with the question, “how much hacking will ASP.NET membership need this time?” So, no matter what I come up with, I hope to roll it into a re-usage package.

TicketDesk 3 Dev Diary – EF Migrations & TD2 Upgrades – Part 2

Fixing MigrationsIn part 1, I outlined TicketDesk’s database management requirements, and how I hoped to use EF Migrations to meet them.

In this part, I will describe how I setup EF Migrations for new databases or TicketDesk 2 upgrades.

Migrations for a new TD3 database:

Ignoring TD2 upgrades, my first mission was to enable migrations for fresh TicketDesk 3 installations.

Bizarrely, Visual Studio 2012 did not include any tooling to support EF 5 migrations –odd since it has tooling for other EF 5 features. Developers interact with EF Migrations through the Package Manager Console; poor man’s tooling, but at least the commands are straight forward and well documented.

The Enable-Migrations command successfully created a migrations folder, generated the initial migration files, and added a migrations configuration class. As I mentioned before, there are database objects you can’t define with data annotations or the fluent API, so the next step was to edit the initial migration to include those pesky default constraints.

Once I had the initial migration finished, I was able to execute it from the PM console to produce the new database. The command line tools are fine for developers, but I also want the web application to initialize the database and run the migrations for me.

EF includes a migrations aware database initializer (MigrateDatabaseToLatestVersion), so I switched my app_start to use it. This also worked fine and the web application will make a fresh database on startup when it needs to.

Great! I’d be done if it weren’t for the need to support TicketDesk 2 databases upgrades.

Migration internals and limitations:

Before I get into how I Migrations work for TicketDesk 2 upgrades, I need to expalin a little about how migrations work internally, and point out some of its limitations.

From the developer’s viewpoint, EF Migrations are pretty simple. The initial migration is just a regular class that inherits from DbMigration. It has two method overrides, one for migrating up (containing create commands), and one for migrating down (containing drop commands). There are two other files associated with the initial migration. These track the parent/child relationships between migrations, but their main purpose is to manage the all-important model hash.

Initial-Migration

The hash is a compressed version of the entire code model at the time the migration was created. When the migration is run, the hash lands in the _MigrationHistory table in the database. This is how EF knows which migration was last used on the DB, and what the model looked like at that time.

The important thing to understand is that EF Migrations never look at your physical database schema. When EF generates a new migration, it does so by combobulating a model from the old hash value. Then it combobulates another model based on the hash of the current code. With before and after models in memory, it uses a complicated diff routine to figure out what needs to be changed to make the models match.

Since the real DB schema isn’t involved in migrations, you can modify the physical database manually, at your own peril, without migrations needing to know about it. In addition, you can modify the statements in the migration class in any way you want; EF will assume that whatever you changed will still result in a database that supports the storage needs of your EF model. Your physical database can also contain other tables and objects that are not related to the model being managed by EF.

The most glaring oversight in the design of EF 5 Migrations is that the _MigrationHistory table can only handle migrations for one model. If you have two different models and DbContexts, you cannot enable migrations on both unless they target different physical databases. If you try to share the same database, the migrations will overwrite each other’s records in the _MigrationsHistory table.

EF 6, currently an Alpha release, will fix this issue with a feature called “multi-tenant migrations”, but I’m not keen on using alpha bits just now.

TicketDesk 2 – Legacy Migration

While I’ve been careful to make sure the initial TD3 database is the same as a TD2 database, there are still minor differences. EF should run a different initial migration against an existing TD2 database. I will call this the “legacy migration” from here out. The legacy migration will just make a few alterations to eliminate the remaining schema differences.

I’ve already talked about how model hashes work, so here is why it matters… Each migration keeps track of its parent migration, which is how EF knows the correct order in which to run them. It is vital that both initial migrations use identical hashes, and have the same migration ID; otherwise, EF will get confused when I add new migrations later, and the linear migration path will be broken.

It took some experimentation to get two initial migrations to co-exist peacefully. Since both must share the same ID, class name, and hash value, it was simplest to put them in different assemblies –TicketDesk.Legacy for the legacy migrations, with regular migrations remaining in the TicketDesk.Domain project.

The legacy migration is a complete forgery, so I didn’t generate it with the PM console commands. Instead, I copied the initial migration files from TicketDesk.Domain into the legacy project. I then changed their namespaces, and gutted the up/down methods. This results in a legacy migration that does nothing, but when executed will add a _MigrationHistory table, and set the migration ID and hash values.

In order to run the legacy migration, EF will need a DbContext and migration configuration class. So, I made a LegacyDbContext. There is no actual model for this project, and we will never generate additional migrations for it, so the LegacyDbContext is empty, except for the constructors. It exists only so EF can open a connection to the database and run this one migration. I made a copy of the standard configuration file too, modifying its namespace and pointing it at the LegacyDbContext.

Legacy-Migration

The last step was writing custom commands into the legacy migration class’s “up” method to take care of those last few schema differences. For completeness, I also added appropriate commands to the “down” method that restores the database to the stock TD2 schema.

At this point, I was able to use the PM console commands to run either the legacy migration, or the regular initial migration, and both work as expected. Looking at the databases with the schema compare tool, there were only cosmetic differences between a fresh DB and an upgraded one. Perfect!

In the next part of this series, I will setup an initializer to selectively run the appropriate initial migration at runtime, and explore a few more gotchas with the EF 5 migrations framework.

TicketDesk 3 Dev Diary – EF Migrations & TD2 Upgrades – Part 1

Sandhill Crane Migration by Serge MelkiTicketDesk 3 may be starting from a fresh code-base, but it is not a new application. There are organizations using TicketDesk in the real world. New releases of TicketDesk have always supported in-place database upgrades, and later versions automated them. I plan to continue this tradition in TicketDesk 3, but would rather not need to roll my own upgrade code, or rely on hacked up T-SQL scripts.

How Migrations Help:

EF code-first is highly capable of generating a well structured SQL database from a pure-code driven model –most of the time. But, EF’s standard database initializers only handle two operations; drop a database, and generate a new database. When you make a model change, EF can automatically drop the DB, and then create a new one that reflects the current model.

This is all fine when developing locally, and is acceptable when deploying a new installation to production. But once you have a production database that real people are using, it is a bad idea to drop and re-create it to deploy a new version of the code.

Customers often dislike having their production databases dropped.

This is where EF Migrations come in. The migrations framework allows EF to modify an existing database in-place, keeping existing data intact. Customers often prefer this approach.

EF make some assumptions about how you will use migrations. It expects a linear, ordered evolution of your database, starting from some “initial state” and ending with the “current state” of your model. The first migration creates a base-line database, then other migrations modify that base-line to reflect any model changes that come along later.

What TicketDesk needs from EF Migrations:

New TicketDesk 3 installations work fine with the normal linear pattern expected by EF migrations. The initial migration will create the database, and fill it with tables. Since I was careful to start with a model that closely resembles TicketDesk 2, the initial migration will create a complete, and reasonably mature, database schema. The TD3 model shouldn’t experience the kind of rapid churn you’d see when building a brand-new product. Still, some model changes will be necessary as work on TD3 progresses, and additional migrations will be needed. The standard design of EF Migrations fully supports this pattern of model evolution.

In my earlier post, I mentioned that there are some database objects that EF can’t automatically generate from data annotations, or the fluent API –default constraints being the main culprit. TicketDesk 3 could get by without default constraints, and the POCO classes could just handle setting defaults in code. But I want my DB to be as similar to the stock TD2 DB as possible. Fortunately EF Migrations does allow you to add real default constraints as part of the migration routine, though you have to manually edit the code to add them.

When upgrading from a TD2 database things gets more complicated. The linear pattern EF migrations expect is not sufficient for upgrades. EF should not attempt to run the regular kind of initial migration against an existing TD2 database. Such a migration would try to create DB objects that already exist, and would fail.

When upgrading, EF will need to run a different initial migration routine –one designed only to modify the DB where necessary to match the normal TD3 schema. Since I was careful to ensure that the two schemas are nearly identical, all the special migration has to do is clean up a few minor variations (e.g. mark an FK as “cascade on delete”, create a non-clustered index, etc).

In theory, once the special migration runs, the old database should be functionally identical to a regular TD3 database. No matter which initial migration routine runs, the same set of subsequent migrations will be able to upgrade the initial DB to reflect model changes that occur later in development.

So that’s what I want from EF Migrations. In the next post, I’ll try to make it happen.