Monday, April 6, 2015

Core Frameworks Don’t Work (most of the time)


I work as a consultant, mostly for large software development organizations at public companies that maintain large systems that handle millions of transactions each day.  My typical engagement is at least 1 year, no longer than 2 years.  It’s long enough that I’m able to dig deep but it’s short enough that I get exposed to many different organizations.  This gives me a unique and informed perspective on how software development works at a fair number of .Net shops.  I’m kind of like Marlin Perkins, able to observe the natural behavior of cutting edge software development organizations in the wild.  

One interesting ritual that I’ve seen practiced over and over is rite of THE CORE FRAMEWORK.  Some organizations actually go as far at to call it THE FRAMEWORK.  Others are content to simply call it core.  Regardless of what you call it, it’s usually a terrible idea that will cripple the productivity of your organization.

A point of clarification

I want to be clear. This post is about company-wide core frameworks.  That’s a common framework that is used as the foundation for multiple applications across the company.  That is not the same as an application core library.

An application core library is usually just the business layer for a single application that can be shared across different assemblies at the application layer.  For example, the SquareHire core library is used for the SquareHire public website, the SquareHire ATS (webapp), the SquareHire admin app (another web app), the SquareHire processor, and the SquareHire webapi. All of these are different assemblies that share a common data and service backend.  Together they make up the SquareHire application so it makes sense that they would share a common SquareHire core library.

A company wide Core Framework is something different.  That’s when a company has multiple applications that do not share a common data and service backend.  They may have nothing to do with each other.  However, the company wants to standardize software architecture across the company so the decision is made to build every application on top of a common, company-wide Core Framework.  Often these Core Frameworks are named after the company, like AcmeInc.core.  These company-wide core frameworks force a common architecture and code base on apps that do NOT share a common data and service backend. These are the frameworks this post is about.

How does it happen?

A group of programmers build an application.  The application works, the company grows, and it hires more developers.  As time goes on the more features are added to the application by developers who really don’t understand the original designs. They just kind of bolt code on that works in the way they’re used to doing things.  The new code doesn’t follow the patterns established by the original designers, but it does accomplish the short term goal.  Besides, the new approach is probably better than the original design anyway.

More time goes on.  The company becomes even more successful which means the app is supporting more users.  It’s now maintained by a completely new group of programmers, nobody’s really sure who the original developers were.  Each new wave of developers implements features using what they feel are the best practices.  The new features hang off the original application like tumors.  Growths that are powered by the original organism, but refuse to cooperate with it.

Things go on this way. The company is now so successful that it is developing multiple applications, or maybe they’re acquired and merged with another organization and their software projects.  However it happens, the point is that we now have a situation where the organization has multiple teams maintaining multiple applications.  Each application has a different architecture and is becoming increasingly more difficult to maintain as new generations of programmers keep bolting stuff on that doesn’t work in the way the original app was designed.

This is the moment when somebody has the bright idea “You know what would make our software development efforts much more efficient and would make all of our applications work much better… if we standardized on a single perfect architecture and made everybody in the entire organization adopt it.  In fact, we should create a set of Core libraries, and then every new project will be built on top of them.”

The theory of company-wide Core

On the surface, a Core Framework makes a lot of sense.  There are some good arguments for standardizing and implementing a common framework across your organization.  Here’s a few of them.

1. New development should be faster/easier
New projects will be much easier to get off the ground. They won’t need to write plumbing code like persistence, they’ll get everything they need from the “Core”.

2. Managing development resources (people) should be easier
If you standardize it should be easier for developers to move between teams.  They’ll already be familiar with the architecture. 

3. The smartest guys can guide us
Since the “Core” will be designed by the architect (or architects) it will embody the best possible architecture for the organization and it will enforce a set of best practices that will be implemented by less capable programmers who might otherwise go astray. 

I may have overdone the sarcasm on that last one but the point is that creating a core really does seem like a good idea on paper.

The reality of company-wide Core

The theory sounds good, but the reality doesn’t work out that way.  Here’s some real world observations pulled from my time at real organizations that were in various stages of implementing Core libraries.  Keep in mind that the architects and developers at these organizations are some of the smartest that I’ve ever worked with.  The problem definitely was not a lack of skill or programming experience.

1. New development should be faster/easier
The Core may actually get the initial phase of a new project off the ground a bit quicker, but it doesn’t last.  Quickly developers find that the Core brings with it limitations as well as benefits. It’s not long until the developers of the new application are scheduling time with the architect and the Core team to try and figure out how Core can be changed to accommodate the needs of this new application.  Did you catch that?  Not only does core not meet the needs of the new app, the new application actually winds up driving changes to Core. This happens for every application that consumes it.  What’s really been created is a massive cross-application dependency.  Nothing brightens my day like pulling the latest build of core and discovering that my entire application is now broken because some other team decided that we needed a better way to manage transactions or the DI container or any of the other things that find their way into these core libraries. 

2. Managing development resources (people) should be easier
This benefit actually does happen to some extent. Different apps in the organization will be structured in a similar way so that it’s easier for developers to move between them.  Unfortunately this commonality comes with negative side effects.  

The worst effect is that software development becomes uniformly slow and difficult.  The Core is a cross application dependency that forces the decisions made by one team onto every other team in the organization.  This also means that when your team needs a change made to core, it’s not going to happen quickly.  You’re going to have to wait for buy in from the architects and other team leads.

I’ve also seen Core negatively effect organizations’ ability to hire. Core usually goes hand in hand with a dramatic increase in complexity.  I’ve heard managers express that it’s incredibly difficult to find programmers and once they do it takes 5 or 6 months for them to be productive because the app architecture is just so complex.  Does the Core have to be that complex? No, but for some reason it always is… which brings us to number 3.

3. The smartest guys can guide us
The core is indeed designed by the smartest programmers in the organization.  This has to be a good thing right?  Wrong.  There are two big problems here. 

First is the problem of misaligned incentives.  When a group of programmers is working together to ship a product, their incentives are all pretty much aligned.  They want to ship the product.  If the product doesn’t ship it’s a big problem for everyone.  When you take your smartest programmers and you pull them out of that team and tell them their priority is to build the core architecture that all other applications will be built on, you’ve just given them a different set of incentives.  You just told them that it’s not their job to ship product. Their job may be to create beautiful architecture or develop new techniques for modeling business domains, or researching new technologies, or to instruct developers who are shipping products on how they should be implementing the core.  The point is, incentives are no longer aligned.    

The second problem is a phenomenon that I’ve labeled Smart Guy Disease.  This is another thing I’ve seen repeated in multiple organizations.  The basic idea is that the worst mistakes that have the most negative impact on an organization are consistently made by the “smart” programmers not the “dumb” programmers.   If you want more detail, I wrote a whole post on it here


The company-wide core framework is an idea that seems like an obvious win to management and architects.  But be warned.  Here there be monsters.  I have never, not once, seen a company-wide architecture solve more problems than it solved.  I have seen two examples where company-wide frameworks had hugely negative impacts on their respective organizations.  So think twice before starting down that path. 

One alternative I would suggest is to maintain a base application code-base.  But instead of forcing every application to consume it and inherit from it (thus creating cross-application dependencies), just fork it when you create a new app.  That way you have a solid starting point, but you’re free to make changes without worrying about how those changes will affect other apps in your organization.

Sunday, January 11, 2015

You might be using TFS

If your dev computer has 500GB of disk space taken up by 237 branches of the same code even though 99% of the files in those branches are identical…  you might be using TFS.


If you spend 45 seconds watching a little spinning icon each time you rename a file… you might be using TFS.


If your supervisor requires that you log your time in 15 minute increments and track those increments against work items in your current sprint… you might be using TFS.


If the workflow for tracking a bug fix requires a simple 27 click process that includes setting values in 3 different status fields and manually reassigning the work item and still doesn’t map to your actual process… you might be using TFS.


If the most productive team in your company manages their sprints in Excel instead of the company standard ALM software – and you wish you were too… you might be using TFS.


If creating a new branch requires more than 10 seconds… you might be using TFS.


If you ARE NOT CONFUSED when you hear people say things like “It took me 5 minutes before I was able to switch to disconnected mode” or “I spent a full day trying to fix a corrupted workspace”… you might be using TFS.


If you’ve ever heard the phrase “Yeah it’s kind of inefficient and it has some rough spots and the constantly connected model is a pain, but it’s sooooo much better than Source Safe”… you might be using TFS.


If you’ve ever lost 2 days trying to fix history after a bad merge… you might be using TFS


If your company has ever sent all of the devs home because the source control server was down and nobody could work… you might be using TFS. 


I find it tragic that so many .Net teams use what I consider to be the “worst of breed” solution just because it’s made by Microsoft.  TFS is a cumbersome, poorly designed system that extracts a productivity tax on every developer, every day that they use it.  The constantly connected model alone is reason enough for me to never use this tool.  So why do people use it if it’s so bad? I think it comes down to 3 unfortunate facts of life:

1. SCM (Source Control Management) and ALM (Application Lifecycle Management) software is typically selected by managers, not by the developers who will need to use those systems.

2. Microsoft makes TFS.  It’s a safe choice.  Nobody is going to get fired for picking the solution made by Microsoft.

3. Most of the teams using TFS have either never used another SCM, or were previously using Microsoft Source Safe, the only SCM so horrible that it actually makes TFS look good by comparison.

So the next logical question is “Ok Mr. Complainer, if TFS is so bad what do you use instead?”

I’m glad you asked.  In my opinion there is only one choice, git. A git repository hosted on either Bitbucket (from Atlassian) or GitHub, and managed using  SourceTree (also from Atlassian) is such a gloriously simple and low friction solution that I’ve never found cause to use anything else.  If you find yourself in the position of picking an SCM, do yourself a favor and try the git/SourceTree/(GitHub or Bitbucket) solution first and see if it meets your needs.

ALM/sprint planning/bug tracking is another matter.  There are lots of SAAS apps that provide this functionality and I find all of them easier to use than TFS.  GitHub itself has some basic ALM functionality that will be good enough for most small teams.  There are also 3rd party SAAS apps that plugin to GitHub or Bitbucket to provide additional features.  Huboard is one such app that provides a Kanban board for your issues on GitHub.  The bottom line is try something simple and see if that works for you before moving on to a more complex solution like Jira or (shudder) TFS.

One last thing.  I know some people will read this and say “You’re totally out of touch, you can use git as your source control within TFS, get all the git advantages, and still get all the great TFS vertical integration.”  That might be true. I’ve never used TFS with git and I’ve never seen it used in any of the TFS shops that I’ve worked with. That’s worth noting. I’ve heard a lot of people talk about TFS with git, but I’ve never actually seen it implemented in the wild. So, it might be the land of milk and honey. It might fix every problem I’ve ever had with TFS. I doubt it, but it is possible. 

To conclude, I don’t have an informed answer to the TFS with git question because what I’m doing works great and I’m not willing to invest another iota of my precious time into the sinkhole of productivity that is TFS.   I think the great philosopher Samuel Jackson (Pitt) from Pulp Fiction said it best. “Sewer rat might taste like pumpkin pie, but I wouldn’t know, because I wouldn’t eat the filthy mother-scratcher.”

Friday, May 23, 2014

SafeSubstring Extension Method

Extension methods are a great way to keep you’re view code tight and readable.  I always include a UIExtensionMethods class in my projects that contains extensions for the HtmlHelper and for my View Model classes.

The Problem

I often want to restrict the max number of characters that will display in the UI.  If a field has 2000 characters, I just want to show the first 100.  Easily done right?  Just use a substring like so:

<div> @Model.JobSummary.Substring(0, 300) <a href="#" class="empub_feed_readmore_link">read more...</a> </div>

The problem is that there are a number of things that will make Substring throw errors.  If JobSummary is null, error.  If JobSummary has less than 300 characters, error.  I can check for these things in my view code, but that’s going to make my view look messy and I’ll have to copy and paste that code every time I do a substring in a view.

A Quick Solution

Anytime I have view code that needs to be used more than once, or even hurts the readability of my view, I extract that code into an extension method.  Since I want a version of Substring that I can run in a view without throwing an error, I call the method SafeSubstring.

// SafeSubstring public static MvcHtmlString SafeSubstring(this HtmlHelper htmlHelper, string text, int start, int stop) { if (string.IsNullOrEmpty(text) || text.Length < (start + stop)) { return new MvcHtmlString(text); } return new MvcHtmlString(text.Substring(start, stop)); }

Now my view code looks clean, and I have another tool in my toolbox that can be used next time

<div> @Html.SafeSubstring(Model.JobSummary, 0, 300) <a href="#" class="empub_feed_readmore_link">read more...</a> </div>

Sunday, March 30, 2014

Smart Guy Disease


As a consultant I’ve had the opportunity to see first-hand how software development works at some of the most cutting-edge .Net shops in the Denver area.  One of the things I’ve seen over and over is a phenomenon I call “Smart Guy Disease”.

The basic concept is this.  The biggest, most destructive mistakes are always made by the “smart” programmers, not the “dumb” programmers.   The smarter the programmer, the greater the capacity for catastrophe.

It seems counter-intuitive, but it’s true.  Think about it for a minute.  “Dumb” programmers…. actually I have to stop a moment because I don’t really like the term dumb programmers.  One of the reasons I like software development is that I consistently get to work with smart people.  Even programmers who aren’t up on the latest technologies and are just showing up to do their job every day tend to be very intelligent people.   However, I think we all have worked with people who just weren’t up to the task.  They consistently ship code rife with bugs, you have to code review every line they commit with a fine tooth comb, they just aren’t very good at programming and they consistently produce bad code.  Bad programmers might be a better term, but I’ll keep using “dumb” because it contrasts well with smart.

Anyway, “dumb” programmers do produce crappy code, but they almost never make the kind of massive mistakes that cripple an entire organization’s ability to ship software.  They may implement features that perform poorly, but they don’t deploy code that destroys performance on a global scale.   Nope, when you want to cause that kind of damage, you need a smart guy. 

I’ve seen it.  It happens.  The smartest guys in an organization are the ones that make the truly big mistakes.

Shipping architecture instead of software

The first mistake I’ve seen is shipping architecture instead of software.  Companies take the smartest guys in the organization and tell them that they’re architects now and they are no longer responsible for shipping an actual application. Instead they will create a company-wide enterprise architecture, or Core Framework,  that other app development teams, who do ship production applications, will be required to build on top of.

I’ll go into more detail on why this is a bad idea in a later post, but for now just know that in this situation the incentives of the architects and the app developers are not aligned.  The architects’ priority is to create architecture, something that implements cutting edge technologies, looks great in diagrams, is impressive when presented to C level executives, and if it’s overly complex… well that just demonstrates how smart they are.   The app development team’s priority is to produce code that performs well, is easy to maintain, and ships on time.  The architects really have no idea if their designs are easy to implement or perform well until they are implemented by the app team.  Not a good way to ship code.

Overly complex solutions

Another common problem, that I alluded to above, is overly complex solutions.  You might be tempted to dismiss this one as no big deal.  Don’t.  This is probably the most common and most damaging mistake that I’ve seen in the real world.  Smart programmers usually like to show how smart they are.  You can’t really do that if you produce code that is easy to understand and maintain.  Why implement a simple solution when you can do something clever instead? 

So what’s the cost of extra complexity? If you look only at the first 9 months of an application’s lifecycle then the answer is “not much”.  However, if your application is successful and has a lifecycle that extends beyond the first year then that extra complexity becomes a tax that you pay over and over, every time someone touches that code base.  That’s the real problem.  The price for additional complexity isn’t just paid by that initial developer, it’s paid again and again by every developer who touches that code for the entire lifetime of the application.  It’s far too common for smart developers to ship a new product with needlessly complex architectures, then declare the project a success and move on, leaving a maintenance nightmare for the “average” developers who come after them.  

Complexity also leads to brittle code.  We all understand intuitively that the more complex something is, the more likely something will go wrong and it will break. That’s bad enough by itself. Now think a year down the road when that complex system is being modified by a developer who doesn’t necessarily understand the intent of the original “smart” programmer. Now you’ve got a high probability that something will get broken.  

The crazy thing is that teams sometimes look at the complexity of their code as a badge of honor.  If you ever catch yourself saying something like “We are so cutting edge that we can only hire the best developers, and even then it takes them a good 4 months before they’re productive with our architecture”…. maybe you should ask yourself if that’s really a good thing.  

Smart on paper, disaster in reality

One last mistake, solutions that look great on paper but are a complete disaster when implemented in the real world.  The best example I’ve seen of this was when a company decided that they wanted to rewrite their ASP.Net WebForms applications so that they could be managed with a 3rd party CMS.  The idea was that the WebForms pages were mostly made up of .ascx controls anyway, so it should be possible to make every .ascx responsible for loading and saving it’s own data, then publish that data to and coordinate with the other controls on the page at runtime.  These individual controls could then be composited together in any combination by designers or business people using the CMS. There was some kind of limitation with the CMS that required all of the heavy lifting to be done in the browser with JavaScript and Ajax.  Did I mention this was WebForms code with update panels and all that nonsense. Remember how fun it was to do real JavaScript and manage DOM Ids browser-side with update panels?

To be fair, it’s been a while and I don’t really remember the details.  I just remember that the big brains decided this made sense, and I’m sure it did make sense in a high-level planning meeting.  But, every hands-on-keyboard developer who worked with those apps immediately recognized that this was a terrible idea.  Every developer who actually shipped code could see that even if a miracle happened and this idea worked, it would make the code much, much harder to maintain and it would slow development to a crawl.  It did by the way.

How to avoid smart guy disease

So what’s the point?  Is this just a rant because I hate smart people?  No of course not.  I like smart people and I like working with smart people.   The point is that it’s possible, even common, for the smartest developers, who should be our greatest asset, to instead create massive problems.  But it doesn’t have to be that way.

I have 2 simple rules that can help prevent smart guy disease from crippling your organization.

Rule 1.  Make sure your smartest developers are always in a role where they are shipping production code. 

Most of the problems mentioned above happened when the smartest developers became disconnected from the reality of building and maintaining production software.  That disconnect messes up the incentives and puts your smartest programmers at odds with the programmers who actually ship code.

Instead, keep your best and brightest developers embedded in teams that are shipping production applications.  Let them be a team leader or a team member.  Give each application team their own architect. Or, if they can’t be a permanent team member, at least let them be a temporary resource embedded inside the team. They can work hands-on-keyboard with the other developers to actually implement their architectural designs.  That kind of hands on experience will help everyone involved and will keep those incentives aligned.

Rule 2. Value simplicity. 

Make simplicity a cultural value, a goal that every developer strives to attain.  Make it clear that simple solutions are valued and needlessly complex solutions won’t be tolerated.  Keep a close eye on any developers who want to use multiple layers of base classes, Unity Interceptors, or other techniques that are used to make things happen automagically.  

Never, ever boast that your architecture is so complex that only the very best developers can work on it.  Instead your boast should be that your architecture is so well designed that any junior developer can come in off the street, understand the basics in a day or two, and be productive within the week.

Wednesday, December 19, 2012

NHibernate WCF Error: HTTP request context being aborted by the server??

I’ve been ambivalent toward ORMs for quite some time now, but I’m really starting to dislike NHibernate in a special way.  My experience has been that ORMs lead to a lot of wasted time troubleshooting code problems that are really ORM problems.  Here’s one example.

I recently got this mysterious and unhelpful error message on an ASP.Net MVC3 web app that was pulling data via a WCF web service, which in turn was using NHibernate for persistence.

An error occurred while receiving the HTTP response to http://localhost:8080/JobsService/ws. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details.

Ok, my service endpoint was definitely using HTTP and none of that other stuff was happening either.  So what was the problem?  NHibernate lazy load proxies.

Let’s say I have a Company entity that contains a child collection of type List<Job> as shown in the diagram below.  So if we look at the Company entity, all of the fields (Name, State, City) exist in my data table, except for the Jobs.  Jobs represents a relation between my Company and Job tables and to get Job data NHibernate has to query the Job table.


At this point it’s important to note that I’m using NHibernate, all of my entity properties are virtual, and I have lazy loading enabled.  So what happens when I run a query to get a Company? What data does NHibernate populate?  It’s going to populate all of the data that’s in the Company table (Name, State, City) , but it’s not going to populate the Jobs collection.  It’s going replace that virtual property with a proxy that will execute a query to get Jobs only when that property is accessed. 

I actually think this lazy loading proxy technique is pretty neat and it’s technically impressive.  Unfortunately it’s also the root cause of a lot of application errors.

So the lazy loading proxy works great when we’re accessing an entity directly in our web app while it’s still has the context of the NHibernate session, but what happens when we try to return our Company over WCF and WCF tries to serialize that Jobs property that hasn’t been lazy loaded yet?  That’s right, you get the ambiguous error message above. 

The fix is simple.  You just need to eager load the Jobs. You can do this any number of ways.  Here’s how to do it in a criteria query.

return session.CreateCriteria<Company>()
            .Add(Restrictions.Eq("Id", _Id))
            .SetFetchMode("Jobs", FetchMode.Eager);

Now that we’re eager loading the Jobs there’s real data there instead of the lazy load proxy, WCF has no problem serializing the Jobs, and our mysterious ambiguous error goes away. 

So, I hope this saves someone a little time, and please remember, friends don’t let friends use ORMs.

Wednesday, July 18, 2012

The problem with ORMs

pigdogOver the last 8 years I’ve done a lot of thinking and experimenting with different persistence models. If you’d asked me 6 years ago what a persistence layer should look like I would have whole heartedly suggested an ORM like nHibernate.  Linq to SQL came along and initially looked promising but I quickly soured on it in favor of Entity Framework.  Then I actually tried to use Entity Framework in a production app and quickly found that in spite of the initial query writing efficiencies, it caused problems in other areas and the net result was that it didn’t save me any time at all but did add limitations and complexity to my app. 

So what was the problem?  As I spoke with more developers I started to hear the same complaints, even though some were using EF, some were using nHibernate, others were using LinqToSql.  I heard the problem stated best by a guy I met at the speaker’s dinner for the Rocky Mountain Tech Trifecta.  I can’t remember the guy’s name but his words stuck with me. I was telling him that I would never use EF again if the choice was up to me.  He said it’s not that he has a problem with Entity Framework, it’s that he’s done with ORMs.  I think that’s really the heart of it.  There’s a fundamental problem with ORM as a pattern.  It just doesn’t adequately solve the problem and it ads significant complexity and inefficiency to an application.

I recently ran across a couple of old blog posts that make for interesting reading, Is OR/M an anti pattern? by Ayende, and ORM is an anti-pattern by Laurie.  I’m hearing this line of thought more often as more developers use EF and other ORMs in production apps and then have a couple of years of living with the consequences.

So it’s easy to knock something, but what’s the alternative? For me it’s been moving to a simpler data mapper instead of a full on ORM.  I first wrote my own simple data mapper, but since then I’ve discovered by Sam Saffron and Marc Gravel.  On the surface Dapper works almost identically to the mapper I wrote, but under the covers it’s way better and much more efficient. So I’ve been porting all of my old code over to use Dapper. 

The results so far have been fantastic.  Sometimes I have to write more code that I would have to write with a full-on ORM, but I find that I can make major application changes (relatively) quickly and easily and I have never once spent hours struggling with my persistence architecture trying to understand why I’m erroring out on an update for an object graph, or troubleshooting lazy load issues, or select N+1 issues, or thinking to myself “If only this was SQL I would know exactly how to do this”.  Instead it just works, exactly the way I expect it to, and in the end that saves a lot of time and effort.