TicketDesk 3 Dev Diary – Angular it shall be!

which wayIt has been a while since I updated everyone on TicketDesk 3. I took a break to wrap up several other projects and to search for a new day job, but now I’m back to working on TicketDesk 3 again.

I haven’t been idle though. I spent much of this spring working with Asp.net identity, OWIN authentication middleware, and all the federated identity goodness — Azure AD, Azure Graph API, ADFS, Azure AD Sync, WsFederation, oAuth, OpenID Connect, etc.

I now have a decent grasp of what’s going on in the world of modern authentication and identity. There is a LOT happening in this space right now, and the Asp.net stack is right there on the bleeding edge of it all.

Unfortunately, documentation and guidance on how all the new security pieces fit together in real world apps is sparse, but I’ve learned enough now to be comfortable that I can get TD3 to handle multiple authentication scenarios.

Over the last several weeks, I’ve also been working deeply with Angular.js. I’ve dabbled in SPA’s in the past a bit, mostly on the Durandal, knockout.js, and breeze stack –which the current TD3 Alpha uses. This space has also been moving fast, and Angular in particular has been gaining traction like crazy.

For me, the impetus for examining Angular in more depth was the announcement that Durandal’s principal author had joined the Angular team, and that Durandal will merge with the next version of Angular (2.0). I really liked Durandal, but Angular is what all the cool kids are using –it just makes more sense to go with the tide, rather than stick with a soon-to-be-obsolete framework

After working with Angular.js a bit, I decided to go ahead and move TD3 over now, before the UI gets any larger or more complex.

So far, I’m enjoying Angular. It is a more complex platform than Durandal, so getting my head around it required significant re-training. But I’m finding that it is an amazingly productive platform.

As before, I’m basing the early TD3 platform on the courses from John Papa, and his Hot Towel Angular packages. Hot Towel made a good starting point for Durandal, and it makes an even better one for Angular. This time though, I’m not sticking as close to the Hot Towel provided UI bits. Instead, I’m working with a much more advanced theme from the wrap-bootstrap project.

Here’s the short-term plan:

  • Start with the Hot Towel Angular packages.
  • Mix in a gutted version of the new theme –leaving just the parts I intend to use.
  • Hook up Asp.Net text resources to i18Next and Angular’s internationalization filters.
  • Integrate client bearer token security with asp.net identity and OWIN middleware.
  • Setup sever-side Breeze and the Breeze.js library to move the data around.
  • Put the SignalR stuff back in.
  • Tackle production js/css minification
  • Clean it up, and package it as a starter kit
  • Build out the rest of TD3 on the new platform.

I have a couple of upcoming projects with platform needs, so this base-line starter kit will serve as a common ancestor . If it’s good enough, I might even make it a public nuget package or something.

I’ll commit code to github and codeplex as soon as I have the platform in a usable state, and can afford to buy the extended license I need to redistribute the theme I’m using.

 

TicketDesk 3 Dev Diary – Update: One AspNet and Aspnet.Identity

Since most of what I’m working on now is being done in my private repository, I wanted to give everyone a quick update on TicketDesk 3′ s progress.

I’m still working on TD3, but late in the fall it became apparent that I needed to wait on the RTM version of Visual Studio 2013 and the new Asp.net Web API 2 and Identity bits. Now that this stuff has all been released, and most of the dependencies have caught up, I have resumed work on TD3.

The main challenge is incorporating the new Aspnet.Identity framework. I am thrilled that Microsoft finally replaced the old security providers, but the transition to Aspnet.Identity is not all roses. The new framework is not as well documented as I’d like, and guidance on advanced uses is still thin. It is also fairly complex framework that requires a good bit of manual coding, especially when used in conjunction with a SPA front-end. Fortunately for me, Yago Pérez Vázquez has created a template project called DurandalAuth that does exactly what I’ve been trying to do with TicketDesk…. combine the bleeding edge versions of Aspnet.Identity and Web API 2 with the bleeding edge versions of Durandal, Breeze, and Bootstrap.

In fact, his template is so good, that I’m pretty much building TicketDesk 3 on top of his template instead of just porting the authentication stuff into my existing TD3 project… and this is why the project hasn’t been pushed into the public repository just yet… it’s a new project that doesn’t yet have all the features from the old one.

There are a few things about the DurandalAuth template that I’m not so sure about; the use of StructureMap instead of Ninject for IoC, and fact that he’s layered the back end with a custom repository and unit of work pattern… but overall, the design is generally the same as what I had been designing for TD3; except that he’d also implemented the new asp.net identity bits. The template also includes some SEO stuff that isn’t relevant to TicketDesk 3, though I may leave it there in case people want to make use of it.

At present, I’m in the process of combining the new project with the code I’ve already written for TD3, and adapting the design to TD3’s particular needs (internationalization for example). This will take a few more weeks, but once I’m done I will be able to push the new project to the public GitHub repository for everyone else to look at.

The technology stack for TD3 is now complete; and includes the following:

  • Asp.net Web API 2
  • Durandal 2 & Breeze
  • i18Next
  • Aspnet.Identity
  • Bootstrap 3
  • SignalR 2

The only major design element that I’ve yet to work out completely is using ACS and/or ADFS security. The identity implementation for Web API 2 uses a different mechanism (organization accounts) for integrating with ACS and ADFS; so I’ll have to find a way to smash the two options together; and provide enough internal abstractions to where either configuration is possible with minimal additional code.

TicketDesk 3 Dev Diary – MEF, IoC, and Architectural Design

TicketDesk 2 and TicketDesk 3 have some key architectural differences. Both enforce a strict separation of concern between businesses and presentation layers, but there are major architectural differences within each layer. In this installment, I’d like to talk about how the back-end architecture will evolve and change.

TicketDesk 2 – Decoupled design:

The most significant technology that shaped TicketDesk 2’s class library design was the use of the Managed Extensibility Framework (MEF). The use of MEF in TicketDesk 2 was not about modularity, at least not in a way that is relevant to business requirements. TicketDesk 2 was never intended to support plug-ins or dynamic external module loading. I used MEF for two reasons; I was giving test driven development (TDD) another shot, and I had planned to write a Silverlight client for TicketDesk 2.

MEF was originally built by the Silverlight team. It had a lot of potential for other environments, but didn’t play well with MVC back then. It took some dark magic and hacking to just make it work there. MEF is an extensibility framework first, but an IoC container only by accident. While MEF can do the job of an IoC container, it wasn’t particularly good in that role.

As an extensibility framework, MEF actually has more in common with require.js than traditional server-side IoC frameworks. As a Silverlight technology, the primary purpose was to enable clients to download executable modules from the server on demand when needed. This is exactly what require.js does for JavaScript in HTML applications. The truly interesting thing is that TicketDesk 2 did not use MEF in this way at all. Asp.Net MVC is a server-side environment following a request-response-done type execution flow. Deferred module loading isn’t relevant in that kind of environment. TicketDesk used MEF only for its secondary IoC features — runtime composition and dependency injection.

Considering the difficulty in getting MEF working, and the fact that there are better IoC frameworks for MVC, I should have scrapped MEF in favor of Ninject –which has made me very happy in dozens of other projects. I stuck with MEF partly because it would pay off when I got to the Silverlight client, and partly because I liked the challenge that MEF presented.

Sadly, I was only three weeks into development on TicketDesk Silver, the Silverlight client, when Microsoft released Silverlight’s obituary. I had two other projects under development with Silverlight at the time, so that was a very bad summer for me.

The modular design of TicketDesk’s business layer is mostly about testability. EF 4 was quite hostile to unit testing, so I did what everyone else was doing… I wrapped the business logic in unit-of-work and repository patterns, and made sure the dependencies targeted abstract classes and interfaces. If you want to get all gang-of-four about it, the service classes in TD2 are more transaction script than unit-of-work, but it gets the same job done either way. This gave me the level of testability I needed to follow a (mostly) TDD workflow.

One thing I have never liked about heavy unit testing, and TDD in particular, is having to implement complex architectures purely for the sake of making the code testable. I’ll make some design concessions for testability, but I have a very low tolerance for design acrobatics that have nothing to do with an application’s real business requirements.

TicketDesk 2 walks all over that line. I dislike that there are a dozen or more interfaces that would only ever have one (real) concrete implementation. Why have an interface that only gets inherited by one thing? I also dislike having attributes scattered all over the code just to describe things to an IoC container. Neither of those things make TicketDesk work better. It just makes it more complex, harder to understand, and harder to maintain.

On the flip-side, I was able to achieve decent testability without going too far towards an extreme architecture. The unit tests did add value, especially early in the development process –They caught a few bugs, helped validate the design, and gave me some extra confidence.

If you noticed that the current source lacks unit tests, bonus points to you! My TDD experiment was never added to the public repository. I was pretty new to TDD, and my tests were amateurish (to be polite). They worked pretty well, and let me experience real TDD, but I didn’t feel that the tests themselves made a good public example of TDD in action.

TicketDesk 3 – Modularity where it matters:

A lot has changed for the better since I worked on TicketDesk 2.

Some developers still write their biz code in a custom unit-of-work and repository layer that abstracts away all the entity framework stuff; which is fine. But when EF code-first introduced the DbContext, it became much friendlier towards unit testing. The DbContext itself follows a unit-of work pattern, while its DbSets are a generic repository pattern. You don’t necessarily need to wrap an additional layer of custom repository and unit-of-work on top of EF just to do unit testing anymore.

I plan to move most of the business logic directly into the code-first (POCO) model classes. Extension methods allow me to add functionality to any DbSet<T> without having to write a custom implementation of the IDbSet interface for each one. And the unit-of-work nature of the DbContext allows me to put cross cutting business logic in the context itself. Basically, TD 3 will use something close to a true domain model pattern.

As for dependency injection, the need to target only interfaces and abstract types has been reduced. An instance of a real DbContext type can be stubbed, shimmed, or otherwise mocked most of the time. In theory, I should be able to target stubbed/shimmed instances of my concrete types. If I find the need to target abstracts, I can still refactor the DbSets and/or DbContext to inherit custom interfaces. There still isn’t a compelling need to wrap the business logic in higher layers of abstraction.

In TicketDesk 3, I will not be using a TDD workflow. I love unit testing, but am traditionally very selective about what code I choose to test. I write tests for code that will significantly benefit from them –complex and tricky code. I don’t try to test everything. Using TDD as a design tool is a neat thought process, but I find that design-by-test runs counter to my personal style of design. I can easily see how TDD helps people improve their designs, but I personally tend to achieve better designs when I’m coding first and testing later.

When I do get to the need for dependency injection, I plan to run an experimental branch in TicketDesk 3 to explore MEF 2 a bit further. I think they have fixed the major issues that made MEF 1 hard to use in web environments, but it is almost impossible to find good information online about MEF 2. The documentation, when you can find it, is outdated, contradictory, and just plan confusing. What I have found suggests that MEF 2 does work with MVC 4, but still requires some custom infrastructure. What I don’t know is how well it works.

With the need for dependency injection reduced, few compelling extensibility requirements on the back-end, and no plans to do heavy unit testing, I am more inclined to go with Ninject. They care enough to write top-notch documentation, and it was designed explicitly for the purpose of acting as an IoC container… which is the feature set TicketDesk actually needs.

TicketDesk 3 Dev Diary – SignalR

One of the overall design goals of TicketDesk since version 1 has been to facilitate near-frictionless, bi-directional communication between help desk staff and end-users. Tickets should evolve as a natural conversation, and the entire history of those conversations should be right there in the ticket’s activity log. TicketDesk has done a reasonably good job in this area, but SignalR presents an opportunity to take this idea to a whole different level.

The goal behind the SignalR library is to give the server a live, real-time channel to code running on the client. The server can push notifications whenever things change, and that information is available to the user immediately. The techniques that SignalR use to achieve this are not entirely new, but have historically been difficult to implement.

TicketDesk 3 uses breeze on the client, and breeze tracks all the entities it has retrieved from the server. Knockout is used to bind those entities to the UI for display. The beauty of this combination is that Knockout automatically updates the UI anytime the entities in breeze change.

With SignalR, the browser can listen in the background for updates from the TicketDesk server. When the server notifies the client that a ticket has changed, the client can then choose to fetch the new data in the background, and update the local copy being tracked by Breeze… and Knockout will automatically refresh the display to show that updated data to the user.

The best thing about SignalR is that it is trivially easy to setup, and with the combination of Breeze and Knockout it is super simple for the UI to respond intelligently.

As a proof of concept, I have coded up a simple SignalR hub that will tell all connected clients when a ticket changes (and what the ID of the changed ticket is). The client will check to see if it is tracking a ticket with that ID, and if so it will automatically fetch a new copy of the ticket from the server. Anything on the screen that is bound to that ticket will automatically update to show the changes. This was not only very easy to implement, but it seems to work very well.

I then took it a step further, and coded up several special routines for the ticket list view. Not only does it update the tickets displayed on screen, but it also responds intelligently to changes in the number of open tickets, or changes of the order of the tickets within the list.

This list view, as currently implemented in the alpha, is a paged list showing 5 items on screen at a time. Because the list is sorted by the last updated date, anytime a ticket is updated the order of items in the list changes too. If a ticket is closed or re-opened, the number of items will grow or shrink change. Pager buttons may need to be disabled or enabled, items on the current page may change as tickets are shuffled around, and the current page might not even be valid anymore if the number of open tickets shrinks enough.

With very little effort, I was able to code up the list view that dynamically responds to real-time changes on the server, and keeps itself current without the user ever needing to explicitly refresh the screen.

I plan to use the set of capabilities around SignalR to make the entire application behave in near real-time. The Ticket activity log will behave like a real-time chat conversations, lists will automatically adjust as things change, and notifications will appear to keep the user informed.

If you want to take a look at the early alpha version, I have published a TicketDesk 3 demo on Azure. I can’t promise that it will be stable, and it certainly isn’t a complete end-to-end implementation. But feel free to play around with it.

To see the SignalR behavior in action, just open the site in two browsers at the same time. Make changes to a ticket in one, and watch the other browser update itself.

TicketDesk 3 Dev Diary – Hot Towel

toweliconFor TicketDesk 3, what I most hope to achieve is an improvement in the overall user experience. Since I wrote TicketDesk 2, much has happened in the JavaScript world. New JavaScript frameworks have matured, and enable deeper user experiences with much less development effort than ever before. TicketDesk is a perfect candidate for Single Page Application (SPA) frameworks, so all I had to do was pick a technology stack and learn to use it.

I have decided to start from the wonderful Hot Towel SPA by John Papa. Hot Towel is a visual studio project template that combines packages from several different client and server frameworks. It includes Knockout.js for UI data-binding, Durandal for routing and view management, and Breeze for talking to the ASP.NET Web Api backend.

My main reasons for choosing Hot Towel are:

  • It is a complete end-to-end SPA Template.
  • It is well documented.
  • The components it relies on are reasonably mature, and well maintained.
  • There are good sample applications built on Hot Towel.
  • John Papa published an excellent, and highly detailed, video course for Hot Towel at Pluralsight.
  • It is very easy to learn and use.

One of the disappointments when switching from server-side Asp.Net to a SPA framework is that the UI is pure JavaScript, HTML, and CSS. It makes almost no use of MVC controllers or views, which always makes me feel like I’m wasting valuable server capabilities. A SPA does make heavy use of Asp.Net Web Api for transferring data, but the UI leaves all that wonderful Asp.Net and Razor view engine stuff behind.

Once I learned by way around Hot Towel, I was surprised to find that working with Knockout, Durandal, and Breeze on the client is much easier than working with Asp.Net on the server. I’m no fan of JavaScript as a language, but the current crop of JavaScript frameworks are truly amazing.

Now that I’ve learned my way around Hot Towel’s various components, I’ve been able to develop a fairly advanced set of UI features very quickly. The current UI is very raw and only provides a primitive set of features, but it has already exceeded my initial expectations by several orders of magnitude.

If you want to take a look at the early alpha version, I have published a TicketDesk 3 demo on Azure . I can’t promise that it will be stable, and it certainly isn’t a complete end-to-end implementation. but feel free to play around with it.

TicketDesk 3 Dev Diary – Thoughts on Security

hunts point, the bronx, new yo... by andre dos santosIn this post, I’m brainstorming new approaches to handling security in TicketDesk 3, and reviewing the limitations of TicketDesk 2’s old approach.

Security in  TicketDesk 2:

TicketDesk 2 shipped with support for either active directory, or local security. Local security via the built-in membership providers worked reasonably well for most deployments, but the AD option has always been problematic.

Windows authentication, in conjunction with the WindowsTokenRoleProvider, works well enough for the basic question, “who are you?”, but windows authentication severely limits in the amount of information you can obtain about the user. You get the windows account name and a list of the user’s AD groups –and that’s pretty much all you get.

Some features in TD required more than that. It needed display names and email addresses, and it needed to list all users in some AD groups; information the built-in providers are incapable of obtaining.

To support these requirements, TicketDesk 2 used a complicated background processes for AD concerns. It maintains a local cache of AD data to reduce the number of queries, and moved AD query processing off of the user servicing request threads.

Even with the abstractions in System.DirectoryServices, querying AD is painfully obtuse. In addition, response times for AD queries are often measured in the tens of seconds –even in single domain environments. You can’t reasonably query AD in real-time without bringing the entire application to a standstill. The custom module alleviates these issues, but was a  complete nightmare to code and maintain.

The other significant problem has been the difficulty of extending the old system to support more features, such as custom roles and permissions. This is why TicketDesk still uses hard-coded roles with pre-defined permissions –I simply never had the time or energy to re-factor the AD side of the system.

Security technologies of the present:

I’ve played a bit with the SimpleMembership and Universal Providers, which have replaced the original SqlMembership providers. ASP.NET’s security system is still based on the same provider framework that shipped with .Net 2.0 back in 2005. Sadly, the framework is a fundamentally flawed system, and in need of a complete redesign.

In recent years, there have been two attempts to modernize the ASP.NET providers, but neither really addresses the underlying problems of the core framework itself. Both are just extensions layered on top of the original infrastructure.

Universal Providers are an evolutionary upgrade of the original SQL providers from ASP.NET 2.0. They mainly just move data access into Entity Framework, which allows them to work with any database back-end supported by EF. Otherwise, there isn’t a much of a difference between them.

SimpleMembership was created for ASP.NET WebPages, but has become the officially recommended replacement in MVC projects as well. This is the default security provider in the newer MVC project templates that come with Visual Studio 2012. As I wrote in another post though, this system has some deep design flaws of its own. You almost have to learn how to hack up a customized version of SimpleMembership just to make it useful in any real-world scenario.

The good news is that both SimpleMembership and Universal Providers include oAuth support backed by the popular DotNetOpenAuth libraries. So, while flawed and build on a shaky foundation, at least they have been modernized.

Traditional windows authentication itself has not changed in any significant way, and there have been no official replacements for the windows security providers that I know of. ASP.NET still uses the same windows provider options that shipped in .Net 2.0, all of which are spectacularly terrible.

The major action for Active Directory authentication has been with Active Directory Federation Services. This is an STS token server that exposes AD to clients that support WS-federation. The Windows Identity Foundation (WIF), originally introduced in WCF, was pulled directly into the core of the .Net framework with the 4.5 release. This provides the infrastructure ASP.NET needs to leverage federated authentication and authorization. So, if you setup federation in ASP.NET you can easily connect it to ADFS to authenticate your AD users. Best of all, ADFS can share any information stored in AD.

In Windows Azure land, you have the Access Control Service (ACS), which is also a WS-Federation token service. From ASP.NET, connecting to ACS is similar to connecting to ADFS. The real advantage of ACS is that it can act as an aggregator and proxy for any number of other external authentication sources. For example, you can connect your ACS to your domain’s ADFS service and to oAuth providers.

The new identity features in .NET 4.5 also incorporate claims based identity as a pervasive and normative feature used throughout the entire framework. You don’t have to use WS-federation in order to use claims based identities, since all identities are now claims based.

Ideas for TicketDesk 3:

For TicketDesk 3, I hope to support the following authentication options:

  • local user accounts (forms with user registration)
  • oAuth (sign-in with google, facebook, twitter, etc.)
  • Windows (AD or local system accounts)
  • WS-Federation (ADFS, Azure ACS, etc.)

My challenge is to find a way to bring all these pieces together in TicketDesk 3 in a sane way.

I want TicketDesk’s business logic to make use of claims based identity for its authorization decisions. Since claims are baked in, this seems to be the most sensible way of doing things. The key will be to map all users, regardless of how they are authenticated, to a common set of claims that TicketDesk understands. The business logic can then rely on having those claims attached to every user’s identity.

For authentication, I have little choice but to rely on existing mechanisms. I am not a crypto expert, and have no intention of rolling my own password and user verification code. But, to support multiple authentication sources, I will likely need to roll my own module to intercept user requests, and dynamically route them to the appropriate pre-built authentication mechanism as needed at runtime.

Each of the possible authentication systems will produce a ClaimsIndentity, but the structure and content of those claims varies. Some authentication generates very few usable claims, while others may contain all the claims TicketDesk could ever need. To bridge the gap, TicketDesk will need an identity transformer. This component will map incoming identities, obtained during authentication, to a TicketDesk identity with all the claims TicketDesk needs. Since some claims cannot be obtained during authentication, TicketDesk will need to obtain any missing data from the user, or an administrator, then store and manage that data locally.

As you can tell, this is all in the early conceptual stages. I’m doing a lot of experiments and research, but I’m at the point where I have a good grasp of most of the pieces that I’ll need to smash together.

I have several other projects with similar security requirements, and I’ve grown tired of approaching each new project with the question, “how much hacking will ASP.NET membership need this time?” So, no matter what I come up with, I hope to roll it into a re-usage package.

TicketDesk 3 Dev Diary – EF Migrations & TD2 Upgrades – Part 2

Fixing MigrationsIn part 1, I outlined TicketDesk’s database management requirements, and how I hoped to use EF Migrations to meet them.

In this part, I will describe how I setup EF Migrations for new databases or TicketDesk 2 upgrades.

Migrations for a new TD3 database:

Ignoring TD2 upgrades, my first mission was to enable migrations for fresh TicketDesk 3 installations.

Bizarrely, Visual Studio 2012 did not include any tooling to support EF 5 migrations –odd since it has tooling for other EF 5 features. Developers interact with EF Migrations through the Package Manager Console; poor man’s tooling, but at least the commands are straight forward and well documented.

The Enable-Migrations command successfully created a migrations folder, generated the initial migration files, and added a migrations configuration class. As I mentioned before, there are database objects you can’t define with data annotations or the fluent API, so the next step was to edit the initial migration to include those pesky default constraints.

Once I had the initial migration finished, I was able to execute it from the PM console to produce the new database. The command line tools are fine for developers, but I also want the web application to initialize the database and run the migrations for me.

EF includes a migrations aware database initializer (MigrateDatabaseToLatestVersion), so I switched my app_start to use it. This also worked fine and the web application will make a fresh database on startup when it needs to.

Great! I’d be done if it weren’t for the need to support TicketDesk 2 databases upgrades.

Migration internals and limitations:

Before I get into how I Migrations work for TicketDesk 2 upgrades, I need to expalin a little about how migrations work internally, and point out some of its limitations.

From the developer’s viewpoint, EF Migrations are pretty simple. The initial migration is just a regular class that inherits from DbMigration. It has two method overrides, one for migrating up (containing create commands), and one for migrating down (containing drop commands). There are two other files associated with the initial migration. These track the parent/child relationships between migrations, but their main purpose is to manage the all-important model hash.

Initial-Migration

The hash is a compressed version of the entire code model at the time the migration was created. When the migration is run, the hash lands in the _MigrationHistory table in the database. This is how EF knows which migration was last used on the DB, and what the model looked like at that time.

The important thing to understand is that EF Migrations never look at your physical database schema. When EF generates a new migration, it does so by combobulating a model from the old hash value. Then it combobulates another model based on the hash of the current code. With before and after models in memory, it uses a complicated diff routine to figure out what needs to be changed to make the models match.

Since the real DB schema isn’t involved in migrations, you can modify the physical database manually, at your own peril, without migrations needing to know about it. In addition, you can modify the statements in the migration class in any way you want; EF will assume that whatever you changed will still result in a database that supports the storage needs of your EF model. Your physical database can also contain other tables and objects that are not related to the model being managed by EF.

The most glaring oversight in the design of EF 5 Migrations is that the _MigrationHistory table can only handle migrations for one model. If you have two different models and DbContexts, you cannot enable migrations on both unless they target different physical databases. If you try to share the same database, the migrations will overwrite each other’s records in the _MigrationsHistory table.

EF 6, currently an Alpha release, will fix this issue with a feature called “multi-tenant migrations”, but I’m not keen on using alpha bits just now.

TicketDesk 2 – Legacy Migration

While I’ve been careful to make sure the initial TD3 database is the same as a TD2 database, there are still minor differences. EF should run a different initial migration against an existing TD2 database. I will call this the “legacy migration” from here out. The legacy migration will just make a few alterations to eliminate the remaining schema differences.

I’ve already talked about how model hashes work, so here is why it matters… Each migration keeps track of its parent migration, which is how EF knows the correct order in which to run them. It is vital that both initial migrations use identical hashes, and have the same migration ID; otherwise, EF will get confused when I add new migrations later, and the linear migration path will be broken.

It took some experimentation to get two initial migrations to co-exist peacefully. Since both must share the same ID, class name, and hash value, it was simplest to put them in different assemblies –TicketDesk.Legacy for the legacy migrations, with regular migrations remaining in the TicketDesk.Domain project.

The legacy migration is a complete forgery, so I didn’t generate it with the PM console commands. Instead, I copied the initial migration files from TicketDesk.Domain into the legacy project. I then changed their namespaces, and gutted the up/down methods. This results in a legacy migration that does nothing, but when executed will add a _MigrationHistory table, and set the migration ID and hash values.

In order to run the legacy migration, EF will need a DbContext and migration configuration class. So, I made a LegacyDbContext. There is no actual model for this project, and we will never generate additional migrations for it, so the LegacyDbContext is empty, except for the constructors. It exists only so EF can open a connection to the database and run this one migration. I made a copy of the standard configuration file too, modifying its namespace and pointing it at the LegacyDbContext.

Legacy-Migration

The last step was writing custom commands into the legacy migration class’s “up” method to take care of those last few schema differences. For completeness, I also added appropriate commands to the “down” method that restores the database to the stock TD2 schema.

At this point, I was able to use the PM console commands to run either the legacy migration, or the regular initial migration, and both work as expected. Looking at the databases with the schema compare tool, there were only cosmetic differences between a fresh DB and an upgraded one. Perfect!

In the next part of this series, I will setup an initializer to selectively run the appropriate initial migration at runtime, and explore a few more gotchas with the EF 5 migrations framework.

TicketDesk 3 Dev Diary – EF Migrations & TD2 Upgrades – Part 1

Sandhill Crane Migration by Serge MelkiTicketDesk 3 may be starting from a fresh code-base, but it is not a new application. There are organizations using TicketDesk in the real world. New releases of TicketDesk have always supported in-place database upgrades, and later versions automated them. I plan to continue this tradition in TicketDesk 3, but would rather not need to roll my own upgrade code, or rely on hacked up T-SQL scripts.

How Migrations Help:

EF code-first is highly capable of generating a well structured SQL database from a pure-code driven model –most of the time. But, EF’s standard database initializers only handle two operations; drop a database, and generate a new database. When you make a model change, EF can automatically drop the DB, and then create a new one that reflects the current model.

This is all fine when developing locally, and is acceptable when deploying a new installation to production. But once you have a production database that real people are using, it is a bad idea to drop and re-create it to deploy a new version of the code.

Customers often dislike having their production databases dropped.

This is where EF Migrations come in. The migrations framework allows EF to modify an existing database in-place, keeping existing data intact. Customers often prefer this approach.

EF make some assumptions about how you will use migrations. It expects a linear, ordered evolution of your database, starting from some “initial state” and ending with the “current state” of your model. The first migration creates a base-line database, then other migrations modify that base-line to reflect any model changes that come along later.

What TicketDesk needs from EF Migrations:

New TicketDesk 3 installations work fine with the normal linear pattern expected by EF migrations. The initial migration will create the database, and fill it with tables. Since I was careful to start with a model that closely resembles TicketDesk 2, the initial migration will create a complete, and reasonably mature, database schema. The TD3 model shouldn’t experience the kind of rapid churn you’d see when building a brand-new product. Still, some model changes will be necessary as work on TD3 progresses, and additional migrations will be needed. The standard design of EF Migrations fully supports this pattern of model evolution.

In my earlier post, I mentioned that there are some database objects that EF can’t automatically generate from data annotations, or the fluent API –default constraints being the main culprit. TicketDesk 3 could get by without default constraints, and the POCO classes could just handle setting defaults in code. But I want my DB to be as similar to the stock TD2 DB as possible. Fortunately EF Migrations does allow you to add real default constraints as part of the migration routine, though you have to manually edit the code to add them.

When upgrading from a TD2 database things gets more complicated. The linear pattern EF migrations expect is not sufficient for upgrades. EF should not attempt to run the regular kind of initial migration against an existing TD2 database. Such a migration would try to create DB objects that already exist, and would fail.

When upgrading, EF will need to run a different initial migration routine –one designed only to modify the DB where necessary to match the normal TD3 schema. Since I was careful to ensure that the two schemas are nearly identical, all the special migration has to do is clean up a few minor variations (e.g. mark an FK as “cascade on delete”, create a non-clustered index, etc).

In theory, once the special migration runs, the old database should be functionally identical to a regular TD3 database. No matter which initial migration routine runs, the same set of subsequent migrations will be able to upgrade the initial DB to reflect model changes that occur later in development.

So that’s what I want from EF Migrations. In the next post, I’ll try to make it happen.

TicketDesk 3 Dev Diary – EF Code-First Model

Nowhere to stand, nowhere to hide by Pete Wright Photography, on Flickr

It is often difficult to decide where to start when building a new application, but for TicketDesk 3 the obvious first step is to create the database and entity model.

Until now, the TicketDesk database has followed an unbroken evolution from the original TD1 database to TicketDesk 2.0.3. I built the original database by hand, then used it to generate a Linq to SQL model diagram. When I put TicketDesk 2 together, I just switched to EF 4 entity model diagrams; again, building the model from the existing database.

For TicketDesk 3, I’ll once again switch data access technologies, this time using EF 5’s DbContext with a code-first model (POCO). I could continue with *.edmx models, and generate the DbContext and POCO classes from the diagram, but Microsoft seems to be moving away from *.edmx, and I strongly prefer the code-first approach anyway.

Initially, I want to start TicketDesk 3 with a model as similar to the TicketDesk 2 database schema as possible. Due to the way EF generates T-SQL, there will be cosmetic differences, but I don’t want any significance structural variations –not yet.

To create the initial POCO classes, I was able to leverage the old entity model diagrams. The tooling in VS 2012 allowed me to open the old *.edmx files and add the “EF 5.x DbContext Generator”. This creates T4 templates, which in turn automatically produce generated DbContext and POCO classes based on the diagram. All I had to do was pluck out the generated class files and drop them into the TicketDesk 3 project.

Interestingly, the POCOs created by the generator include the model properties, but not the annotation attributes that identify key fields, required fields, and so on. Normally I’d have to annotate them all by hand, but I found a shortcut for this too. TicketDesk 2 originally used another T4 template, called AutoMetaData, which produces meta classes for all its entities; and those meta classes contain the data annotation attributes. The old EntityContext stuff doesn’t need those annotations, but generating them was useful to enable UI validations in the MVC front-end. So, all I had to do was copy the attributes from the TD2 meta classes, and drop them on the corresponding properties in the new TD3 model classes. Tedious, but easy.

TD 2 also included custom extensions (via partial classes) for a few entities. Mostly these just had a few non-persistent helper properties, and a custom method to two. I simply copied these into new POCOs too, and then made sure they were annotated correctly (adding the NotMapped attribute to most).

By leveraging the work I’d already done in TD 2, I was able to create a nearly complete EF code-first model for TD 3 without writing any code by hand. Sweet!

The last step was to test DB generation from the new model and DbContext.

My web project was created from the stock MVC SPA template that shipped with the “ASP.NET and Web Tools 2012.2 update“. I setup a new localdb connection string in web.config, configured EF to use the DropCreateDatabaseAlways initializer, and then called the Database.Initialize() method on my new DbContext during the application_startup event. Running the web application triggered the database creation without any problems.

I then used the schema compare tool in VS 2012 to compare the new DB against an unmodified copy of the TD2 database. There were a few differences, which I had expected, but structurally it was the same DB that TicketDesk 2 used. The only significant variation was the lack of default constraints in the new database.  EF 5 doesn’t support default constraint definitions via attributes, nor the fluent model builder APIs. Fortunately, EF migrations can handle defaults, so I will solve that problem later on.

The stock TD2 database also included SQL membership, role, and profile tables, which are not a part of  the new entity model. I’ll be addressing the security system pretty soon,  and then I can decide how best to handle the security DB objects.

And that is pretty much it for the initial TD 3 code-first entity model. As I develop TD 3, I will likely make changes, but I can rely on EF to manage the database schema going forward. I’ll need to setup EF migrations, deal with seed data, and add upgrade support for legacy TD2 databases… but those are topics for future posts.

TicketDesk 3 Dev Diary – Source Control and Source Hosting

source code hostingBefore any good project gets further than a mission statement, it needs source control. Typically, I don’t even create a project file without first creating a source repository in which to put it. But I’ve not been that happy with CodePlex, so I’ve decided to move the project to GitHub and switch to the Git source control system.

Source Control Systems

TicketDesk 1 and 2 were hosted at CodePlex using Microsoft’s Team Foundation source control system. After 5 years of trying, I eventually developed a deep and lasting hatred for all things TFS. When CodePlex added support for Mercurial, I had TicketDesk migrated as soon as I could.

I use both Mercurial and Git, but I tend to prefer Mercurial. Both are highly capable systems, but for windows development, the experience around Mercurial is much better. Mercurial installs easily, runs smoothly, is easy to learn, easy to use, and comes with a great GUI out of the box.

In contrast, running Git on windows requires installing a whole mess of junk to bridge the gap between Linux and Windows… and that just gets you to the command-line tools. There are several third party Git GUIs, but most of them are terrible. GitExtensions is the best, though still ugly and counter-intuitive compared to TortoiseHG.

Fortunately, there are great extensions available for Visual Studio no matter which system you go with.

About command lines… I love having the option of a command line, but I hate using a command line as my primary user interface. Command lines are fine for advanced scenarios, automation, or propping up a nice GUI… but for something I interact with as often as my source control system, I want a good GUI!

Repository Hosting

I like CodePlex, but it lags behind BitBucket and GitHub in both features and popularity. CodePlex’s release management features are way ahead of the competition, and the site makes a better end-to-end product hosting service than most of its competitors.

CodePlex elegantly handles product information, downloadable releases, source code, support forums, bug tracking, and documentation. I am completely comfortable sending an end-user to TicketDesk’s CodePlex site to submit bugs or ask questions. In contrast, I would never send an end-user to GitHub or BitBucket to get support for one of my products.

As much as I like CodePlex though, I’ve also had many bad experiences with them, particularly around support and reliability. As an example, my routine request to have TicketDesk converted to mercurial went unanswered for more than 3 months, and it wasn’t handled very well when they did get to it. CodePlex often doesn’t perform very well, though this seems to have improved some recently. CodePlex also isn’t very popular, which makes projects there less discoverable, and people less likely to fork or contribute.

I use Bitbucket extensively for my professional code. Bitbucket lagged behind GitHub for a long time, but in the last year it has improved rapidly. Now, it rivals or exceeds GitHub in most of the areas I care about most. They’ve added support for Git repositories too, though I’ve only used it with Mercurial. I like BitBucket’s pricing model, and it supports organizational team accounts, as well as individual users  –perfect for corporate projects where an individual doesn’t directly own the repositories.

GitHub is crazy popular, and for “social coding” popularity is everything. If you are using Git, and your project is public, then GitHub is the obvious choice. It has amazing features, and sets the bar by which everyone else is measured. The only major complaint I have is the lack of a release management system. GitHub doesn’t make the best full-service project hosting site, but it does a bang-up job with the core mission of source code collaboration. GitHub’s pricing is perfect for open source projects and private repositories, but awkward for some corporate scenarios.

Why Git and GitHub for TicketDesk 3?

    • Git
      • I use Git a lot when working with other peoples’ projects, but I have not used it extensively for my own. I could use more end-to-end experience with it.
      • The typical workflow with Git is perfect for public projects like TicketDesk. Private branches, history culling, and similar features all help keep the public repository clean, and focused only on commits that moved the project forward.
      • Even among the .Net developer community, Git is the more popular option. Using Git will hopefully encourage more people to contribute.
      • Git has better integration with Windows Azure, and I do hope to make TicketDesk 3 an Azure deployable application. This is a minor plus, but worth a mention.
    • GitHub
      • Since I’ve chosen Git, GitHub is the inevitable choice for hosting.
      • Hosting at GitHub should result in higher visibility for the project, which I hope translates into more people discovering TicketDesk.
      • GitHub’s best features are those designed to enable independent programmers to contribute and collaborate on public projects; something I’d like to see happen more frequently with TicketDesk.

What about the CodePlex project?

I do plan to keep the CodePlex project around. I will likely continue to push packaged releases there, especially since GitHub doesn’t do much for release management. I may push occasional source updates to CodePlex when the code reaches major milestones.

The TicketDesk 3 repository is already on GitHub, but is private for now. Once I have a good start, I’ll open it up to the public.