TicketDesk 2.5 – Progress and Developer Notes

I wanted to give everyone an update on the progress of TicketDesk 2.5, and take a few minutes to explain the general architecture that’s starting to take shape over on the develop branch at CodePlex.

Projects and Architecture:

The Database supports any version of SQL Server 2008 or higher, including Azure SQL (but not compact edition). The design of TicketDesk would also fit a document database very well too, so TD may eventually end up having a RavenDB and/or Azure DocumentDB variant.

The Schema is managed by code-first migrations, with the execution of migrations handled by startup initializers and/or on-screen admin tools. It uses multi-tenant, code-first migrations. The identity and the business domains have separate contexts and models, each of which are tenants within the same DB. Similarly, search and email will be separate tenants too.

TicketDesk.Domain is the core business layer. It has no dependencies on web-specific frameworks, azure frameworks, or specific security providers. It does have a dependency on entity framework, and leverages EF as a full scale business service platform, not just as a data access technology.

I call this a “pervasive EF Domain model”. It’s a lot like most domain models you see described in any .net book or tutorial, but without unnecessary abstractions to hide the entity framework components.

The Entities directly contain their own business logic, while DbSets act as generic repositories. Custom repository functions are provided mostly through extension methods. Some extensions are defined in the domain assembly, while those with web specific dependencies are defined within the web application instead –which is why a DI framework or IoC container has’t been necessary within the domain project.

The DbContext is treated as the root business service, and it provides the unit-of-work pattern. Truly cross-cutting business logic will be handled directly by the DbContext, or through extension methods and helper services.

The application doesn’t need a formal DDD style design, but with search and email breaking out into separate services, the design is headed in that general direction. By the time TD 3 is complete, there will likely be a formal service bus mechanism handling communications between separate root business contexts, and this will drive a DDD style eventual consistency pattern.

The first parts of that design will appear in 2.5 with the queue mechanisms that support email and search. It likely will never adhere to the full DDD battery of principals and patterns, but it will borrow heavily from those designs where they make sense.

TicketDesk.Domain.Legacy exists only for conversion of TD2.1 databases… it’s just an isolated container for EF migrations and migration helpers. It has a completely custom initial migration that upgrades a TD 2.1x database to the starting TD 2.5 schema. After that, the regular TD 2.5 migrations from the main domain assembly will bring the database up to the final schema version.

TicketDesk.Search encapsulates all search related functionality. Currently it has local lucene and azure search providers. This could be decomposed into separate assemblies, but I haven’t seen the need yet to go that far with it yet.

Currently, it also fakes a “queue” based mechanism, but before TD 2.5 ships this will be replaced by a more formal queue management system.

TicketDesk.Identity encapsulates all non-web/owin specific security functionality. This is an odd one architecturally, since much of the overall identity system does need dependencies on owin middleware components. Right now, the core EF and aspnet.identity stuff is isolated here, but the web app layers on additional user and role managers.

Much of this is boilerplate identity code borrowed from stock samples, so as I refactor it for TD’s specific needs, I’m thinking seriously about moving all of the identity stuff to this assembly, and letting it have owin dependencies. I’m waiting until I tackle ADAL and/or OpenID Connect (for AD federated security) before I decide for sure, but I don’t want to go as far as to write a custom abstraction layer for this thing.

Either way, the identity stuff will remain decoupled from the business domain assembly entirely.

TicketDesk.Web.Client is the main web application. Nothing special about this design, except that it is an ultra-modern use of the latest MVC stack. It uses Owin/Katana, Aspnet.Idneity, and all the new toys like that.

It makes light use of the Dependency Resolver, with Simple Injector as the underlying DI service. The UI is written to bootstrap 3, with some light jQuery and custom javascript here and there. It does use the jquery unobtrusive ajax stuff for partial view rendering –a concession that makes it easier to port the old UI without having to completely re-invent the entire design.

Mail / WebJob: I haven’t implemented these yet, but email will be split out like search, and will use a DB queue table when running on-prem, and will use Azure Queue Storage when running on Azure. On Azure, mail delivery will be handled by Azure WebJobs.

What I’m working though now, or in the very near future:

Building out the rest of the core UI:

I’m working my way through porting the UI to MVC 5, Razor, and bootstrap 3 now. TicketCenter is close to done, and I’m currently working on the New Tickets screen. Afterwards I’ll get into the main ticket viewer/editor.

The goal is to reproduce most of the current behavior exactly as it exists in TD 2.1. Later, I’ll revisit the behavior to further streamline it. The Viewer/Editor is an command/action design, rather than a view/edit/submit design, so it lends itself better to an MVC style design pattern anyway. There are a few activities that do need some smoothing out, but most of that will wait until TD3.

Main Text Editor:

TicketDesk has alternately used an HTML WYSIWYG editor, or a Markdown editor in past versions. I’ve always intended to support both at the same time, but for some reason never seem to get around to it.

TD 2.5 will offer both options though. I have the PageDown editor (markdown) working in the new ticket editor already, but I also plan to implement a limited html editor –probably SummerNote. The back-end has always supported both kinds of content, so all I have to do there port the current code over from TD 2.1.

There will be admin settings where the admin can choose which editors are enabled, and which is the default. If multiple are available from the settings, each user will be able to set their own preference.

Ticket Center Lists Editor:

TD 2.1’s ticket center is designed around the idea of user tailored lists (my tickets, history, open tickets, etc.). In TD 1, you could customize these lists, and I intended to make that a feature of TD 2 as well. A customization screen appropriate for non-technical users is a big challenge, and TD 2 also had a significantly more complex way of managing settings for these lists, so I never had time to make that happen the way I wanted.

For TD 2.5, we will at least have tools to let admins change the default list definitions, maybe by just letting them edit the raw JSON if nothing else. A full end-user custom list designer would be nice, but it will probably have to wait on TD3.

Images & Attachments:

TicketDesk has traditionally only supported file attachments, but not embedded images via the text editor. It stores those files in the database directly. This makes it very easy to move the database from one server to another, and also simplifies backups as well. While performance hasn’t been a problem, this isn’t the best approach in cloud or hosted deployments… if nothing else, the cost of renting SQL storage space is very high compared to file or blob storage.

TD 2.5 will need a plugable storage provider that can store attachments on the local file system (or file share ), or in Azure Storage Blobs. The legacy database migration feature will need to move existing file data to the storage provider before it removes the old attachments table.

Eventually I hope to have support for popular file storage solutions like dropbox and friends.

The front-end text editors will ideally be able to handle inline image uploads, though I don’t think a full media gallery is really necessary –image re-use almost never happens with help desk type systems anyway. If an uploaded image is bigger than a certain size, it should be swapped for a thumbnail in the editor text, and the full image included as an attachment instead.

Eliminate MVC Areas:

I don’t like what areas do to MVC’s routes and navigation, especially with the newer attribute routing stuff. It was cool back in the days of MVC 2 and portable areas, but has outlived its usefulness in MVC 5.

Right now, I do have a couple of areas setup in the code, but I plan to remove these very soon, then cleanup the action and url links so they don’t have to specify an area as custom routedata each time.

Admin:

TD 2.5 will have a “first-run” setup experience. There is a sort of stub for this already, but this will be replaced by an expanded wizard to walk the admin through setting up everything. Most important is setting up the database, security, email, and search features.

Part of this experience will include directing the admin on how to make a few changes in web.config that can’t be automated easily, like adding custom machine keys and such.

Localization (pre-support):

I’ve already have this worked out in TD3. The UI text will be supplied from resource files, instead of being hard-coded. I haven’t quite gotten into the string-heavy part of the system yet, but as soon as I do, I’ll start moving all of the text to resource files. TD 2.5 itself will not provide a full internationalization experience out of the box, but I want to go ahead and get the code in shape for the features coming in TD3. This will also make it super easy for developers to localize the system themselves.

Thanks to everyone for their support and patience!

Dev Diary – Scaling search for an Azure WebSite

azuresearchAdding real search capabilities to a custom web application is never easy. Search is a complex and deeply specialized area of development, and the tools available to us regular developers are monstrously complex.

With TicketDesk 2, I used the popular Lucene.net library to provide search. Ported from Apace Lucene (Java), this is the core technology that powers almost every popular search service, appliance, and search library on the market.

Once you’ve tackled the initial learning curve, Lucene.net isn’t all that difficult to leverage in a simple system like TicketDesk. It is freakishly fast, super flexible, and is a powerful search solution –not quite Google good, but close enough for most applications.

The problem with Lucene is that the design revolves around indexes stored on a traditional file system. There are 3rd party extensions that let you store the indexes in a database or in the cloud, but internally these all mimic the behaviors of a file system –that’s just how Lucene works.

You can have many components querying an index at the same time, but only one can write to an index at a time. Normally this single-writer limitation isn’t a huge problem. You code your application so it creates just one writer instance, then share it with any components that want to make an index update. As long as you keep things synchronous, it tends to work fine.

And here lies the problem. TicketDesk 2.5 and 3.0 are designed to run at scale, and will ship ready for deployment to the cloud as an Azure WebSite. In this scenario, there can be several instances of the application running at the same time, each needing to write to a single, shared Lucene index.

I spent a full week trying to find a way around the single-writer problem. WebSites in Azure shouldn’t write to their filesystems. Anything written locally is volatile, and vanishes whenever Azure automatically moves the site to a different host. So, I started with the AzureDirectory library for Lucene, which lets you store the search index in Azure blob storage. This works well, and gives Lucene a stable place to store shared indexes in the cloud.

The second problem was keeping multiple web site instances from writing to the index at the same time. Even though the index is in blob storage, Lucene still demands an exclusive write lock. Each websites can see when the index is locked by another writer, but there isn’t a way to know if the lock is legitimate, or an orphaned lock left behind when some other instance went down unexpectedly.

The only easy solution is to make sure there is a separate application to handle all index writes, and that there is only a single instance of that application running. You can scale the websites or other clients, just don’t scale the index writer application.

WebJobs were designed specifically for handling background on behalf of Azure WebSites, so I started there. Each website would queue index updates to an Azure Storage Queue, then the WebJob could come along and process the queue in the background. But WebJobs scale with the websites, so if you have multiple websites, you also have multiple webjobs. Hopefully, in the future MS will give us the ability to scale webjobs independent of the websites they service.

So the only remaining solution would be an old fashioned worker role. They scale independently –or in this case, can be instructed not to scale. This works well, but I just don’t like the solution. Effectively, the worker role ends up being a half-ass, custom search server. It costs a decent amount of money to run a separate worker role instance, plus it complicates the deployment and management of the entire application.

Failing to find a way to continue in Azure with custom Lucene indexes without a centralized search server, I figured I’d just design TicketDesk to take advantage of the existing Azure native solution –Azure Search Services. It is easy to code against (relatively), and there is a free tier that should be suitable for most smaller shops. For larger shops, the costs of a paid Azure Search tier is still reasonable when compared to the costs of a dedicated worker role.

So, out of the box, TicketDesk 2.5 will include at least two search providers; a Lucene provider for on-premise single instance setups, and native Azure Search for cloud deployments. I will eventually add an alternative for on-premise webfarms, and non-azure cloud VMs. In the meantime though, you could still scale in your own data-center by using Azure Search remotely, or stick with Lucene and manually disable the search writer on all but one instance of the site in the webfarm.

One additional note of interest: Azure Search is still in preview, and it doesn’t have an official client library for .Net yet. There are two 3rd party client libraries though; Reddog.Search and Azure Search Client Library. Both are free as NuGet pacakges, but only Reddog.Search has a public open source repository. Also, Reddog has a management portal you can run locally, or install as an Azure WebSite extension.

 

TicketDesk 2.5 – Coming soon!

TicketDesk-2.5While I’ve been working on TicketDesk 3, the code for TicketDesk 2 hasn’t been getting any younger. Since TD3 is still a ways out from a production release, I’ve decided to release a major overhaul of TicketDesk 2 in the meantime. The new TicketDesk 2.5 release will bring the technology stack of TD2 up to the latest version of Microsoft’s Web Platform. Several of the changes will be derived from code developed for TD3.

I am targeting the end of October for a beta version, with a final release in mid to late November (subject to change, as always).

Here are the major changes I have planned so far:

Remove AD Security:

This release will not support direct integration with Active Directory. This was a popular feature in TicketDesk 1 and 2, but it has also been a major problem area as well. Instead, TD 2.5 will only support AD indirectly, via federation (ADFS for example) or through integration with an external identity server (like Azure AD).

Modernized Local Security:

TicketDesk 2.1x still uses the ancient SqlMembership providers that shipped with .Net 2.0 back in 2005. Authorization and identity have come a very long way since then, so TD 2.5 will be upgraded to the newest Aspnet.Identity framework version. It will also provide on-screen management tools to help administrators migrate existing user accounts from their legacy TD 2.1x databases.

UI Changes:

TD2.1x was built on Asp.net MVC 2, with the original Asp.net view engine. This isn’t very well supported by recent versions of Visual Studio, and most developers have long since abandoned the old engine in favor of Razor. I don’t plan many major changes to the general UI’s behavior or appearance, but by re-implementing on Razor I can bring the project into compatibility with Visual Studio 2013 and current Asp.net coding standards.

There will be some minor changes to the UI. I will be removing some of the crusty old JQuery components, and updating the styles to take advantage of newer CSS features that weren’t widely supported when TD2 was first built.

Entity Framework Code-First:

TicketDesk 2.5 will move from Entity Framework 4 database-first models to Entity Framework 6 with a Code-First model. EF Migrations will provide ongoing schema management from here out, including the ability to migrate legacy TicketDesk 2.1x databases. Along with this change, TD 2.5 will also include on-screen database management utilities for migrating legacy databases, seeding demo/test data, and so on.

This refactoring will also bring TD 2.5 in line with the technologies backing TD 3, which will greatly simplify future upgrades.

Eliminate MEF:

The managed extensibility framework has continued to evolve, but it still isn’t a very good choice for Dependency Injection concerns in a web application. Instead, TD 2.5 will use Simple Injector for IoC. Some of the simplifications in the back-end should also reduce the reliance on dependency injection techniques quite a lot.

Improved Email Notifications:

Several improvements to Email Notifications are planned. Most of these are intended to give administrators greater control over how and when TicketDesk sends email notifications. This will include better on-screen testing and diagnostic tools for troubleshooting email related issues.

Multiple Projects:

TicketDesk 2.5 will support multiple projects, which let you handle tickets for different operations, projects, or products in isolation. You will be able to move tickets from one project to another, and can turn off multiple project support  entirely if you don’t need the functionality. I do not know yet if I’ll support user permissions on a per-project basis in this version, but TD3 will certainly provide that functionality.

Watch/Follow Tickets:

Users will be able to watch or follow tickets without having to be the ticket’s owner or assigned user. This will allow these users to receive notifications of changes as the ticket progresses.

Azure Deployments:

TicketDesk 2.5 will be deployable as an Azure WebSite. Currently, there are several issues that make deploying to cloud or farm environments tricky. The biggest are that the Lucene search indexes are stored on the file system, and the old SqlMembership providers are not compatible with the features provided by Azure SQL. These issues are not insurmountable for experienced .Net developers, but deployment to web farms or cloud providers is not currently an out-of-box capability of the system.

To make TicketDesk play well with cloud environments, a pluggable storage provider will be used for any features needing access to non-database storage. When deployed to single instance environments, TD 2.5 will use the file system, but you will be able to reconfigure it for Azure Blob storage when deploying to the cloud. Attachment storage will be moved out of the SQL database to the new storage provider as well.

The only hold-up for Azure SQL is the membership system, but the newer Aspnet.Identity framework fully supports EF migrations and is compatible with any EF data provider –including Azure SQL.

Early pre-alpha code is already committed to the CodePlex repository, and will be updated regularly as work continues. Right now, there isn’t much to see. I’m still working on the basic plumbing for database and identity management, so there are no TD specific user facing features yet. As soon as the back-end is shored up, I’ll start porting in TD specific UI features.

A demo version of the site will be available soon, hosted on Azure. I just have to workout a few minor details related to resetting the database and providing sample seed data, then I can open the demo to the public.

TicketDesk 3 Dev Diary – Angular it shall be!

which wayIt has been a while since I updated everyone on TicketDesk 3. I took a break to wrap up several other projects and to search for a new day job, but now I’m back to working on TicketDesk 3 again.

I haven’t been idle though. I spent much of this spring working with Asp.net identity, OWIN authentication middleware, and all the federated identity goodness — Azure AD, Azure Graph API, ADFS, Azure AD Sync, WsFederation, oAuth, OpenID Connect, etc.

I now have a decent grasp of what’s going on in the world of modern authentication and identity. There is a LOT happening in this space right now, and the Asp.net stack is right there on the bleeding edge of it all.

Unfortunately, documentation and guidance on how all the new security pieces fit together in real world apps is sparse, but I’ve learned enough now to be comfortable that I can get TD3 to handle multiple authentication scenarios.

Over the last several weeks, I’ve also been working deeply with Angular.js. I’ve dabbled in SPA’s in the past a bit, mostly on the Durandal, knockout.js, and breeze stack –which the current TD3 Alpha uses. This space has also been moving fast, and Angular in particular has been gaining traction like crazy.

For me, the impetus for examining Angular in more depth was the announcement that Durandal’s principal author had joined the Angular team, and that Durandal will merge with the next version of Angular (2.0). I really liked Durandal, but Angular is what all the cool kids are using –it just makes more sense to go with the tide, rather than stick with a soon-to-be-obsolete framework

After working with Angular.js a bit, I decided to go ahead and move TD3 over now, before the UI gets any larger or more complex.

So far, I’m enjoying Angular. It is a more complex platform than Durandal, so getting my head around it required significant re-training. But I’m finding that it is an amazingly productive platform.

As before, I’m basing the early TD3 platform on the courses from John Papa, and his Hot Towel Angular packages. Hot Towel made a good starting point for Durandal, and it makes an even better one for Angular. This time though, I’m not sticking as close to the Hot Towel provided UI bits. Instead, I’m working with a much more advanced theme from the wrap-bootstrap project.

Here’s the short-term plan:

  • Start with the Hot Towel Angular packages.
  • Mix in a gutted version of the new theme –leaving just the parts I intend to use.
  • Hook up Asp.Net text resources to i18Next and Angular’s internationalization filters.
  • Integrate client bearer token security with asp.net identity and OWIN middleware.
  • Setup sever-side Breeze and the Breeze.js library to move the data around.
  • Put the SignalR stuff back in.
  • Tackle production js/css minification
  • Clean it up, and package it as a starter kit
  • Build out the rest of TD3 on the new platform.

I have a couple of upcoming projects with platform needs, so this base-line starter kit will serve as a common ancestor . If it’s good enough, I might even make it a public nuget package or something.

I’ll commit code to github and codeplex as soon as I have the platform in a usable state, and can afford to buy the extended license I need to redistribute the theme I’m using.

 

TicketDesk 3 Dev Diary – MEF, IoC, and Architectural Design

TicketDesk 2 and TicketDesk 3 have some key architectural differences. Both enforce a strict separation of concern between businesses and presentation layers, but there are major architectural differences within each layer. In this installment, I’d like to talk about how the back-end architecture will evolve and change.

TicketDesk 2 – Decoupled design:

The most significant technology that shaped TicketDesk 2’s class library design was the use of the Managed Extensibility Framework (MEF). The use of MEF in TicketDesk 2 was not about modularity, at least not in a way that is relevant to business requirements. TicketDesk 2 was never intended to support plug-ins or dynamic external module loading. I used MEF for two reasons; I was giving test driven development (TDD) another shot, and I had planned to write a Silverlight client for TicketDesk 2.

MEF was originally built by the Silverlight team. It had a lot of potential for other environments, but didn’t play well with MVC back then. It took some dark magic and hacking to just make it work there. MEF is an extensibility framework first, but an IoC container only by accident. While MEF can do the job of an IoC container, it wasn’t particularly good in that role.

As an extensibility framework, MEF actually has more in common with require.js than traditional server-side IoC frameworks. As a Silverlight technology, the primary purpose was to enable clients to download executable modules from the server on demand when needed. This is exactly what require.js does for JavaScript in HTML applications. The truly interesting thing is that TicketDesk 2 did not use MEF in this way at all. Asp.Net MVC is a server-side environment following a request-response-done type execution flow. Deferred module loading isn’t relevant in that kind of environment. TicketDesk used MEF only for its secondary IoC features — runtime composition and dependency injection.

Considering the difficulty in getting MEF working, and the fact that there are better IoC frameworks for MVC, I should have scrapped MEF in favor of Ninject –which has made me very happy in dozens of other projects. I stuck with MEF partly because it would pay off when I got to the Silverlight client, and partly because I liked the challenge that MEF presented.

Sadly, I was only three weeks into development on TicketDesk Silver, the Silverlight client, when Microsoft released Silverlight’s obituary. I had two other projects under development with Silverlight at the time, so that was a very bad summer for me.

The modular design of TicketDesk’s business layer is mostly about testability. EF 4 was quite hostile to unit testing, so I did what everyone else was doing… I wrapped the business logic in unit-of-work and repository patterns, and made sure the dependencies targeted abstract classes and interfaces. If you want to get all gang-of-four about it, the service classes in TD2 are more transaction script than unit-of-work, but it gets the same job done either way. This gave me the level of testability I needed to follow a (mostly) TDD workflow.

One thing I have never liked about heavy unit testing, and TDD in particular, is having to implement complex architectures purely for the sake of making the code testable. I’ll make some design concessions for testability, but I have a very low tolerance for design acrobatics that have nothing to do with an application’s real business requirements.

TicketDesk 2 walks all over that line. I dislike that there are a dozen or more interfaces that would only ever have one (real) concrete implementation. Why have an interface that only gets inherited by one thing? I also dislike having attributes scattered all over the code just to describe things to an IoC container. Neither of those things make TicketDesk work better. It just makes it more complex, harder to understand, and harder to maintain.

On the flip-side, I was able to achieve decent testability without going too far towards an extreme architecture. The unit tests did add value, especially early in the development process –They caught a few bugs, helped validate the design, and gave me some extra confidence.

If you noticed that the current source lacks unit tests, bonus points to you! My TDD experiment was never added to the public repository. I was pretty new to TDD, and my tests were amateurish (to be polite). They worked pretty well, and let me experience real TDD, but I didn’t feel that the tests themselves made a good public example of TDD in action.

TicketDesk 3 – Modularity where it matters:

A lot has changed for the better since I worked on TicketDesk 2.

Some developers still write their biz code in a custom unit-of-work and repository layer that abstracts away all the entity framework stuff; which is fine. But when EF code-first introduced the DbContext, it became much friendlier towards unit testing. The DbContext itself follows a unit-of work pattern, while its DbSets are a generic repository pattern. You don’t necessarily need to wrap an additional layer of custom repository and unit-of-work on top of EF just to do unit testing anymore.

I plan to move most of the business logic directly into the code-first (POCO) model classes. Extension methods allow me to add functionality to any DbSet<T> without having to write a custom implementation of the IDbSet interface for each one. And the unit-of-work nature of the DbContext allows me to put cross cutting business logic in the context itself. Basically, TD 3 will use something close to a true domain model pattern.

As for dependency injection, the need to target only interfaces and abstract types has been reduced. An instance of a real DbContext type can be stubbed, shimmed, or otherwise mocked most of the time. In theory, I should be able to target stubbed/shimmed instances of my concrete types. If I find the need to target abstracts, I can still refactor the DbSets and/or DbContext to inherit custom interfaces. There still isn’t a compelling need to wrap the business logic in higher layers of abstraction.

In TicketDesk 3, I will not be using a TDD workflow. I love unit testing, but am traditionally very selective about what code I choose to test. I write tests for code that will significantly benefit from them –complex and tricky code. I don’t try to test everything. Using TDD as a design tool is a neat thought process, but I find that design-by-test runs counter to my personal style of design. I can easily see how TDD helps people improve their designs, but I personally tend to achieve better designs when I’m coding first and testing later.

When I do get to the need for dependency injection, I plan to run an experimental branch in TicketDesk 3 to explore MEF 2 a bit further. I think they have fixed the major issues that made MEF 1 hard to use in web environments, but it is almost impossible to find good information online about MEF 2. The documentation, when you can find it, is outdated, contradictory, and just plan confusing. What I have found suggests that MEF 2 does work with MVC 4, but still requires some custom infrastructure. What I don’t know is how well it works.

With the need for dependency injection reduced, few compelling extensibility requirements on the back-end, and no plans to do heavy unit testing, I am more inclined to go with Ninject. They care enough to write top-notch documentation, and it was designed explicitly for the purpose of acting as an IoC container… which is the feature set TicketDesk actually needs.

TicketDesk 3 Dev Diary – Localization and Internationalization

worldOne of the most frequently requested features for TicketDesk 2 was support for localization. TicketDesk is a stringy application; lots of system generated text that will end up in the user’s face at some point. Localizing TD2 required combing through the source code, line-by-line, translating magic strings by hand.

Clearly, this is not an acceptable approach with TicketDesk 3.

Since localization is thorny, and a weak spot in my own skill-set, I consider it essential to designed for localization as early in the process as possible… and now that the code has gotten to the point where it draws a useful UI, it is time to get started.

In the typical Asp.Net application, localization is mostly just a matter of creating resource files that contain the text translations, then making sure the code only gets text from those resources. There is a lot of support in .Net around localization, cultures, and resource files, so this is pretty easy to do. The only difficult part, for the mono-lingual developer, is getting the text translated into those other languages in the first place.

TicketDesk 3 is a SPA application, which presents a different problem. The UI is mostly HTML and JavaScript, so all that nice .Net localization stuff is unavailable when generating the screens that users will actually see. So the first step was to find a JavaScript library for localization; something that does the same job as .Net resource files. The second challenge was connecting that JavaScript library to the server-side .Net resource files.

Fortunately, there is a fantastic JavaScript library called i18next that fits the bill.

Translations in TicketDesk 3:

i18next follows a pattern similar to server-side .Net resource files. You supply it with json files that contain translated text. Once i18next has the text, it takes care of binding it to the UI via an HTML data-* attributes, or through javascript functions directly. As a bonus, i18next is easy to use in conjunction with knockout’s own UI binding.

TicketDesk performs text functions on the server too, so it still needs resource files, so I wanted to be able to pipe the contents of the resource files to i18next directly, rather than maintaining separate translation files for the server and client. For this, I leveraged Asp.Net Web Api. Configuring i18next to get its json translations from Web Api is simple –just map the URLs it uses to Web Api endpoints.

The Web Api controller itself was a bit more complex. It has to detect the specific language that i18next is requesting, then build an appropriate response in a format i18next can consume. The controller loads a ResourceSet for the requested culture, then loops through the properties to build a dynamic key/value object with all the translated text. Once that’s done, it outputs the dynamic collection as a json response.

i18next has a richer set of capabilities than straight .Net resource files. Resource files are just simple name/value lookups. With i18next, the translation files can have nested properties, replacement variables, and there are features for interpolation (plural forms, gender forms, etc.). These features are available in .Net with some advanced language frameworks, but the basic resource files don’t go that far. Fortunately, TicketDesk only needs the basic set of features, so a flat name/value lookup should be sufficient to get the job done; though it doesn’t leverage some of i18next’s more advanced features.

Localization is more than text translations. TicketDesk also deals with numbers occasionally, and there are some dates too. Fortunately, it isn’t a number heavy application, nor are are there user editable dates or numbers.  The moment.js library easily handles local date display formatting, and numeral.js can handle the couple of numbers.

The main weak point in TicketDesk 3’s localization will be an absence of structural language transformations. Once you get into right-to-left languages and other exotic forms, the physical layout of the entire UI has to change. Sadly, I do not have the expertise to correctly design for such languages.  HTML 5 and CSS 3 do have decent support for this kind of cultural formatting though, so my hope is that anyone needing to localize for these languages can do so themselves without difficulty.

Internationalization:

My intention for TicketDesk 3 was simple localization; the admin would tell the server what language to use, and the system would just generate the UI in that language for all users. I did not initially expect to support dynamic internationalization — the ability to support multiple languages based on individual user preference.

When I got into the details of the i18next implementation though, it quickly became apparent that TicketDesk 3 could easily achieve real internationalization… in fact, internationalization would be about as easy as static localization.

The result is that TicketDesk 3 will be internationalized, not just localized. It will detect the user’s language and dynamically serve up a translated UI for them –as long as resource files exist for their language. If translations for their specific language and culture aren’t available, it will fall back to the best-matching language, or to the the default if no better match exists.

State of the Code:

I have the plumbing for internationalization in place in the alpha already. It auto-detect’s the browser’s language, or you can override it via the query string (e.g. ?setLng=es-MX). Since I don’t speak any other languages, I can’t provide any real translations myself. For the short term, I have created a generic Spanish resource file, into which I copied the English text surrounded by question marks. This isn’t real localization, but it serves to validate that the localization support works correctly.

For dates, I’m using moment.js, so it should adapt to the browser’s language settings automatically, but I haven’t setup moment to use the querystring override yet… I’ll get to that soon though. I’m not doing any number formatting yet, but when I do I’ll implement numeral.js or a similar library.

When TicketDesk 3 gets into beta, and I have the full list of English text strings in resource files, then I will get a native Spanish speaker to help generate a real set of translations. Hopefully, the community will pitch-in to provide other languages too.

If you want to take a look at the early alpha version, I have published a TicketDesk 3 demo on Azure. I can’t promise that it will be stable, and it certainly isn’t a complete end-to-end implementation. But feel free to play around with it. To play with localization, either change your browser’s language to something spanish (es, or es-MX, or similar), or use the querystring override: ?setLng=es

TicketDesk 3 Dev Diary – SignalR

One of the overall design goals of TicketDesk since version 1 has been to facilitate near-frictionless, bi-directional communication between help desk staff and end-users. Tickets should evolve as a natural conversation, and the entire history of those conversations should be right there in the ticket’s activity log. TicketDesk has done a reasonably good job in this area, but SignalR presents an opportunity to take this idea to a whole different level.

The goal behind the SignalR library is to give the server a live, real-time channel to code running on the client. The server can push notifications whenever things change, and that information is available to the user immediately. The techniques that SignalR use to achieve this are not entirely new, but have historically been difficult to implement.

TicketDesk 3 uses breeze on the client, and breeze tracks all the entities it has retrieved from the server. Knockout is used to bind those entities to the UI for display. The beauty of this combination is that Knockout automatically updates the UI anytime the entities in breeze change.

With SignalR, the browser can listen in the background for updates from the TicketDesk server. When the server notifies the client that a ticket has changed, the client can then choose to fetch the new data in the background, and update the local copy being tracked by Breeze… and Knockout will automatically refresh the display to show that updated data to the user.

The best thing about SignalR is that it is trivially easy to setup, and with the combination of Breeze and Knockout it is super simple for the UI to respond intelligently.

As a proof of concept, I have coded up a simple SignalR hub that will tell all connected clients when a ticket changes (and what the ID of the changed ticket is). The client will check to see if it is tracking a ticket with that ID, and if so it will automatically fetch a new copy of the ticket from the server. Anything on the screen that is bound to that ticket will automatically update to show the changes. This was not only very easy to implement, but it seems to work very well.

I then took it a step further, and coded up several special routines for the ticket list view. Not only does it update the tickets displayed on screen, but it also responds intelligently to changes in the number of open tickets, or changes of the order of the tickets within the list.

This list view, as currently implemented in the alpha, is a paged list showing 5 items on screen at a time. Because the list is sorted by the last updated date, anytime a ticket is updated the order of items in the list changes too. If a ticket is closed or re-opened, the number of items will grow or shrink change. Pager buttons may need to be disabled or enabled, items on the current page may change as tickets are shuffled around, and the current page might not even be valid anymore if the number of open tickets shrinks enough.

With very little effort, I was able to code up the list view that dynamically responds to real-time changes on the server, and keeps itself current without the user ever needing to explicitly refresh the screen.

I plan to use the set of capabilities around SignalR to make the entire application behave in near real-time. The Ticket activity log will behave like a real-time chat conversations, lists will automatically adjust as things change, and notifications will appear to keep the user informed.

If you want to take a look at the early alpha version, I have published a TicketDesk 3 demo on Azure. I can’t promise that it will be stable, and it certainly isn’t a complete end-to-end implementation. But feel free to play around with it.

To see the SignalR behavior in action, just open the site in two browsers at the same time. Make changes to a ticket in one, and watch the other browser update itself.

TicketDesk 3 Dev Diary – Hot Towel

toweliconFor TicketDesk 3, what I most hope to achieve is an improvement in the overall user experience. Since I wrote TicketDesk 2, much has happened in the JavaScript world. New JavaScript frameworks have matured, and enable deeper user experiences with much less development effort than ever before. TicketDesk is a perfect candidate for Single Page Application (SPA) frameworks, so all I had to do was pick a technology stack and learn to use it.

I have decided to start from the wonderful Hot Towel SPA by John Papa. Hot Towel is a visual studio project template that combines packages from several different client and server frameworks. It includes Knockout.js for UI data-binding, Durandal for routing and view management, and Breeze for talking to the ASP.NET Web Api backend.

My main reasons for choosing Hot Towel are:

  • It is a complete end-to-end SPA Template.
  • It is well documented.
  • The components it relies on are reasonably mature, and well maintained.
  • There are good sample applications built on Hot Towel.
  • John Papa published an excellent, and highly detailed, video course for Hot Towel at Pluralsight.
  • It is very easy to learn and use.

One of the disappointments when switching from server-side Asp.Net to a SPA framework is that the UI is pure JavaScript, HTML, and CSS. It makes almost no use of MVC controllers or views, which always makes me feel like I’m wasting valuable server capabilities. A SPA does make heavy use of Asp.Net Web Api for transferring data, but the UI leaves all that wonderful Asp.Net and Razor view engine stuff behind.

Once I learned by way around Hot Towel, I was surprised to find that working with Knockout, Durandal, and Breeze on the client is much easier than working with Asp.Net on the server. I’m no fan of JavaScript as a language, but the current crop of JavaScript frameworks are truly amazing.

Now that I’ve learned my way around Hot Towel’s various components, I’ve been able to develop a fairly advanced set of UI features very quickly. The current UI is very raw and only provides a primitive set of features, but it has already exceeded my initial expectations by several orders of magnitude.

If you want to take a look at the early alpha version, I have published a TicketDesk 3 demo on Azure . I can’t promise that it will be stable, and it certainly isn’t a complete end-to-end implementation. but feel free to play around with it.

TicketDesk 3 Dev Diary – Thoughts on Security

hunts point, the bronx, new yo... by andre dos santosIn this post, I’m brainstorming new approaches to handling security in TicketDesk 3, and reviewing the limitations of TicketDesk 2’s old approach.

Security in  TicketDesk 2:

TicketDesk 2 shipped with support for either active directory, or local security. Local security via the built-in membership providers worked reasonably well for most deployments, but the AD option has always been problematic.

Windows authentication, in conjunction with the WindowsTokenRoleProvider, works well enough for the basic question, “who are you?”, but windows authentication severely limits in the amount of information you can obtain about the user. You get the windows account name and a list of the user’s AD groups –and that’s pretty much all you get.

Some features in TD required more than that. It needed display names and email addresses, and it needed to list all users in some AD groups; information the built-in providers are incapable of obtaining.

To support these requirements, TicketDesk 2 used a complicated background processes for AD concerns. It maintains a local cache of AD data to reduce the number of queries, and moved AD query processing off of the user servicing request threads.

Even with the abstractions in System.DirectoryServices, querying AD is painfully obtuse. In addition, response times for AD queries are often measured in the tens of seconds –even in single domain environments. You can’t reasonably query AD in real-time without bringing the entire application to a standstill. The custom module alleviates these issues, but was a  complete nightmare to code and maintain.

The other significant problem has been the difficulty of extending the old system to support more features, such as custom roles and permissions. This is why TicketDesk still uses hard-coded roles with pre-defined permissions –I simply never had the time or energy to re-factor the AD side of the system.

Security technologies of the present:

I’ve played a bit with the SimpleMembership and Universal Providers, which have replaced the original SqlMembership providers. ASP.NET’s security system is still based on the same provider framework that shipped with .Net 2.0 back in 2005. Sadly, the framework is a fundamentally flawed system, and in need of a complete redesign.

In recent years, there have been two attempts to modernize the ASP.NET providers, but neither really addresses the underlying problems of the core framework itself. Both are just extensions layered on top of the original infrastructure.

Universal Providers are an evolutionary upgrade of the original SQL providers from ASP.NET 2.0. They mainly just move data access into Entity Framework, which allows them to work with any database back-end supported by EF. Otherwise, there isn’t a much of a difference between them.

SimpleMembership was created for ASP.NET WebPages, but has become the officially recommended replacement in MVC projects as well. This is the default security provider in the newer MVC project templates that come with Visual Studio 2012. As I wrote in another post though, this system has some deep design flaws of its own. You almost have to learn how to hack up a customized version of SimpleMembership just to make it useful in any real-world scenario.

The good news is that both SimpleMembership and Universal Providers include oAuth support backed by the popular DotNetOpenAuth libraries. So, while flawed and build on a shaky foundation, at least they have been modernized.

Traditional windows authentication itself has not changed in any significant way, and there have been no official replacements for the windows security providers that I know of. ASP.NET still uses the same windows provider options that shipped in .Net 2.0, all of which are spectacularly terrible.

The major action for Active Directory authentication has been with Active Directory Federation Services. This is an STS token server that exposes AD to clients that support WS-federation. The Windows Identity Foundation (WIF), originally introduced in WCF, was pulled directly into the core of the .Net framework with the 4.5 release. This provides the infrastructure ASP.NET needs to leverage federated authentication and authorization. So, if you setup federation in ASP.NET you can easily connect it to ADFS to authenticate your AD users. Best of all, ADFS can share any information stored in AD.

In Windows Azure land, you have the Access Control Service (ACS), which is also a WS-Federation token service. From ASP.NET, connecting to ACS is similar to connecting to ADFS. The real advantage of ACS is that it can act as an aggregator and proxy for any number of other external authentication sources. For example, you can connect your ACS to your domain’s ADFS service and to oAuth providers.

The new identity features in .NET 4.5 also incorporate claims based identity as a pervasive and normative feature used throughout the entire framework. You don’t have to use WS-federation in order to use claims based identities, since all identities are now claims based.

Ideas for TicketDesk 3:

For TicketDesk 3, I hope to support the following authentication options:

  • local user accounts (forms with user registration)
  • oAuth (sign-in with google, facebook, twitter, etc.)
  • Windows (AD or local system accounts)
  • WS-Federation (ADFS, Azure ACS, etc.)

My challenge is to find a way to bring all these pieces together in TicketDesk 3 in a sane way.

I want TicketDesk’s business logic to make use of claims based identity for its authorization decisions. Since claims are baked in, this seems to be the most sensible way of doing things. The key will be to map all users, regardless of how they are authenticated, to a common set of claims that TicketDesk understands. The business logic can then rely on having those claims attached to every user’s identity.

For authentication, I have little choice but to rely on existing mechanisms. I am not a crypto expert, and have no intention of rolling my own password and user verification code. But, to support multiple authentication sources, I will likely need to roll my own module to intercept user requests, and dynamically route them to the appropriate pre-built authentication mechanism as needed at runtime.

Each of the possible authentication systems will produce a ClaimsIndentity, but the structure and content of those claims varies. Some authentication generates very few usable claims, while others may contain all the claims TicketDesk could ever need. To bridge the gap, TicketDesk will need an identity transformer. This component will map incoming identities, obtained during authentication, to a TicketDesk identity with all the claims TicketDesk needs. Since some claims cannot be obtained during authentication, TicketDesk will need to obtain any missing data from the user, or an administrator, then store and manage that data locally.

As you can tell, this is all in the early conceptual stages. I’m doing a lot of experiments and research, but I’m at the point where I have a good grasp of most of the pieces that I’ll need to smash together.

I have several other projects with similar security requirements, and I’ve grown tired of approaching each new project with the question, “how much hacking will ASP.NET membership need this time?” So, no matter what I come up with, I hope to roll it into a re-usage package.

TicketDesk 3 Dev Diary – EF Migrations & TD2 Upgrades – Part 3

dead-birdsIn this last installment, I’ll describe TicketDesk’s custom database initializers, which allow TicketDesk 3 to automatically choose which migration should run on startup. I’ll also confess to a major development blunder!

In my last post, I talked about creating the initial migration for new TicketDesk databases. I used the MigrateDatabaseToLatestVersion<T> initializer in global.asax.cs. This initializer works well for normal EF migration usage patterns, but TicketDesk needs initializers with a bit more spunk.

EF also ships with a feature called “automatic migrations”. I will NOT be talking about this feature here. Automatic migrations are, in my opinion, an abortion-in-progress for anything but the most trivial of applications. I advise you to read up on this feature, and then do your best to avoid it. What I will be talking about here is automatically executing a “code-based migration”.

Great primer on Automatic migrations here, and for Code-base migrations here.

Initializers:

A common technique for customizing database initialization is to write a custom initializer that inherits one of the standard initializers. Sadly, the MigrateDatabaseToLatestVersion initializer is ill-suited for inheritance. There are no overridable methods or extension points to plug into. Instead, I’ll roll my own initializers, implementing IDatabaseInitializer<T>, and take direct control over the migrations process. Fortunately, the EF APIs makes this pretty simple.

Initially, I built one big do-it-all initializer, but later switched to separate initializers that operate independently — in keeping with the SRP.

On Application_Start, TicketDesk will run both initializers. Here is how the process should go:

1) Legacy Initializer Runs
    1a) Check if there is a database
    1b) If a database exists, check the version number (stored in the dbo.settings table)
    1c) If version 2.0.2, run the legacy initial migration (else, do nothing)
2) Regular Initializer Runs
    2a) Run all standard migrations

While the initializers don’t need a mediator, or other coordinating object, the order in which they are invoked is relevant to achieve the desired behavior.

The trick to step 2 is that, if the DB was upgraded in step 1c, then the regular migration will think its own initial migration has already run (remember, both use the same ID and hash value). The regular initializer will still run all additional migrations added later, even if the legacy initializer did the initial migration.

The initializers themselves are rather simple. They instantiate EF’s DbMigrator, which is the code equivalent to using the Package Manager Console’s Update-Database command.

The regular initializer:
public class TicketDeskDatabaseInitializer : IDatabaseInitializer<TicketDeskContext>
{
    private string ConnectionName { get; set; }
    public TicketDeskDatabaseInitializer(string connectionName)
    {
        ConnectionName = connectionName;
    }

    public void InitializeDatabase(TicketDeskContext context)
    {
        var config = new Configuration();
        config.TargetDatabase = new DbConnectionInfo(ConnectionName);
        var migrator = new DbMigrator(config);
        migrator.Update(); //run all migrations 
    }
}
The legacy initializer:
public class LegacyDatabaseInitializer : IDatabaseInitializer<TicketDeskLegacyContext>
{

    private string ConnectionName { get; set; }

    public LegacyDatabaseInitializer(string connectionName)
    {
        ConnectionName = connectionName;
    }

    public void InitializeDatabase(TicketDeskLegacyContext context)
    {
        //if existsing TD 2.x database, run upgrade; creates migration history table 
        if (context.Database.Exists() && IsLegacyDatabase(context))
        {
            var upgradeConfig = new Configuration();
            upgradeConfig.TargetDatabase = new DbConnectionInfo(ConnectionName);

            //this will do nothing but add the migration history table 
            //  with the same migration ID as the standard migrator.
            var migrator = new DbMigrator(upgradeConfig);
            migrator.Update("Initial"); //run just the initial migration
        }
    }

    public static bool IsLegacyDatabase(TicketDeskLegacyContext context)
    {
        // TicketDeskLegacyContext has no DbSets, directly execute select query
        var oldVersion = context.Database.SqlQuery<string>(
                  "select SettingValue from Settings where SettingName = 'Version'");
        return 
        (
            oldVersion != null && 
            oldVersion.Count() > 0 && 
            oldVersion.First().Equals("2.0.2")
        );
    }
}

The Accidental Integration Test:

You will note that the constructor for both initializers requires an explicit connection name, which is used to set the target database for the migrator. This is the result of another oversight in how EF migrations were implemented internally –I consider it an oversight anyway.

The initialize method takes an instantiated DbContext as a parameter, but there is no way to pass that context to a DbMigrator. Instead, DbMigrator always creates a new instance of the DbContext, and it always uses the DbContext’s default constructor. So, in cases where you want to use non-default connections, you must explicitly pass that information into the DbMigrator, otherwise it will use whatever connection your DbContext’s default constructor uses.

I discovered this issue when trying unit test the legacy initializer. My unit test used a custom connection string when it instantiated the DbContext (pointing to a localdb file database in the unit test project). I would run the test, but the DB in the test project refused to change.

Eventually, I discovered that the migrator was making a new DbContext from the default constructor, which in this case was hard-coded to a default connection named “TicketDesk”. This connection string was present in my app.config file, but I had, unwisely, left it pointed at my production database server…  the real TicketDesk 2 database that my company’s IT staff uses!

Yikes!

The test was, indeed, migrating the crap out of my production database!

As a testament to the migrator’s effectiveness, and the similarity between the TD3 and TD2 schemas, TicketDesk 2 didn’t skip a beat. The users never knew their DB had been upgraded, and everything kept working. Later, I manually un-migrated the production database using the PM console commands, which also worked without a hitch.

Lesson learned! Again…

Planting Seeds

Another interesting facet to the design of EF Migrations is how it handles seed data. For migrations, seeding is done through an override method in the migration configuration class. The odd thing about this seed method is that it runs every time migrations are run, even if there are no pending migrations to apply to the database. In TicketDesk’s case, both seed methods will run (from legacy and regular configuration classes).

In my opinion, there are two kinds of “seed” data –there is “default data” which is data that should be pre-set when the DB is created, then there is “test data”, which is used for testing or just pre-loading some example data. EF uses “seed data” in the sense of “test data”.

As a practical example of the difference, TicketDesk has a bunch of default settings that are stored in its settings table. These values are required, or the application fails, but admins can change the default settings later if they choose. This is not, in the sense EF uses the term, “seed data”. It is “default data”, for which EF has no explicit support.

For default data, the best solution seems to be inserting the data during the migration itself –by adding insert operations to the migration class’s up and down methods. Unfortunately, there aren’t any convenient helper methods for doing inserts, so you have to issue raw SQL insert commands as strings.

For “test data”, you can rely on the seed method in configuration, as long as you are comfortable with your data being reset every time the application starts. At least you can use Linq helper methods and your strong-typed entities there.

At this point, I have working database initialization and full migrations support in TicketDesk 3, at least for the core model entities. Security, and security database support, may prove more challenging.