“Can you ask Sam about that? Stacker is his domain”, “I’d rather let Josh look at the router, he wrote it”, “Jon is better versed in associations, send it to him”.
The natural progression of programs is towards the territorial. When a programmer has weaved an intricate web of considerable complexity, others are loathe to enter his lair and he is loathe for them to do so.
This is despite the fact that we all agree that it’s bad for programs to become territorial. When only one or a few people know how to work on something, you get bottlenecks where progress is stunted until the master is ready. You risk the hit-by-a-bus factor where nobody knows how the system works if the master leaves. You ensure the annoyance of stakeholders who can’t understand why another minion can’t fix his urgent problem.
But this problem can’t be solved with a slogan. You can proclaim that “we shouldn’t have territorial parts of our program” until you turn blue, but nothing is going to change until you accept the cost of avoidance.
The first step of acceptance is to recognize that sending someone fresh in to fix a single issue in a complex part of the code is expensive. It’s going to take Pratik five to ten times the effort to fix a single issue in Stacker that it’s going to take Sam. And the odds are that even that is not enough to appreciate the internal coherency of the system, which means that the fix is likely to be a butcher’s job, and Sam will have to rewrite it afterwards anyway.
To broaden the base of knowledge, you’re going to have to let someone else not only spend considerable effort getting up to speed. Then you’re going to have them deal with more than just a quick fix. Let them deal with a raft of issues and let them spend the time of the original creator to learn it all.
To do all that, they can’t do anything else at the same time. That feature you want do is now going to be pushed a few days or a a week out. Until you’re ready to delay things you really want done, it’s fruitless to bemoan that parts of the code base territorial.
Neil Middleton
on 11 Oct 11So, how do you fix it?
Joey
on 11 Oct 11Have you found any processes or strategies to avoid this? Something like pair programming or code review. That way multiple people are aware of these complexities.
Derek
on 11 Oct 11Sounds like something pair programming can help solve…
DHH
on 11 Oct 11Neil, prescription in the second to last paragraph.
Derek, pair programming just hands the territory to the pair. As in “Oh, that’s a Bundler issue. Let’s wait for carlhuda to fix that”.
Agentcurry
on 11 Oct 11As a developer I’ve seen this at every workplace in my career. The best thing I can think of to avoid this is:
1. Have multiple developers working on the same “modules” during the initial implementation and/or maintenance.
2. Have detailed code walkthroughs. This kind-of works, but you have to motivate people to come and you learn much better if you actually work on it yourself.
I dont think there is a way of getting around this withough incorporating a process in your workflow to account for it. In doing so, you will ultimately pay a price in time for the extra steps.
Just my 2 cents…
Anonymous Coward
on 11 Oct 11Code reviews are an ok solution. They are also great for getting a second set of critical eyes looking at implementation decision, in addition to readability and understanding. lastly they are an opportunity to get someone else excited about the code they didnt write.
Making documentation a priority also can help a ton . I’m a fan of plan English readme’s and block comments before methods.
any other solutions ??
MattO
on 11 Oct 11previous comment is mine. forgot to put in my name
Programmer
on 11 Oct 11I’ve come across these 2 types of programmers. Ones that avoid areas that did not create and would rather pass the buck to the creator (about 99%) and ones that are ready to roll up their sleeves and jump into anything (about 1%). It is the 1% that are really in demand right now in the technical industry. They are rare, hard to find and expensive.
rick
on 11 Oct 11See also: Promiscuous Pairing: (hat tip from @patmaddox)
At GitHub, we try to get around this through upfront documentation (README.md files can live in any directory, not just the root), some pairing, and lots of Pull Request code reviews. Important areas of the site usually have several go-to people. We still have a lot of work to go on this, but the situation is better than it was a year ago.
Having a team of developers ready to jump into unknown territory helps. It’s very important for us because the vast majority of the main app was written by the founders. I spent my first month or so knee deep in support requests and fixing every bug in every territory I could find.
Deltaplan
on 11 Oct 11Not to mention the fact that some managers are actually seeking territoriality…
Many times in my career, I’ve seen projects on which the managers are intentionally preventing each developer from working on some part of the program, sometimes even not letting them having access to the corresponding source code. The reason they give for it is simple : if one of the programmers is hired by a competitor, it is preferred that he doesn’t know everything about the software he was working on.
Joe Larson
on 11 Oct 11As long as my code is for my eyes only, I can tolerate a lot of clumsy, hacky code (ugly but working). As soon as I go to share it with someone else, those things become embarrassing and time consuming to explain/document, so I often go on a cleaning spree first! This is a huge benefit to avoiding territorial programming.
On the flip side, territorial programming has huge efficiencies that are often worth the risks.
Neil Middleton
on 11 Oct 11@dhh But the prescription doesn’t really sound like a fix, only dealing with it head-on. Surely there must be a way that this overhead isn’t necessary via documentation or the like. Whilst these all require an up front effort, there must be ways of mitigating the overall time spent on the problem.
I’m out of ideas though.
Shubham Bansal
on 11 Oct 11I think we as programmers are responsible for making programs territorial. Our components become our merit badges, our trump card for job security because no one has a clue how it works. It’s easier to delegate an issue to someone because that’s his/her component but its hard as hell to dig into their code and figuring it out.
Here is my suggestion to this problem. Give people sometime every week to look into a component of their choice (written by someone else) and see if they can make it better (or find bugs). You could associate some kind of reward with this if you like. Besides that, we could write code that is well documented and set up code review sessions to get everyone on the same page. Pair programming session can also help.
Graham Peel
on 11 Oct 11This post is painfully, beautifully accurate.
Yet I do agree with Joe Larson that sometimes, the efficiencies gained by that one person managing the life of the project can be substantial. Just make sure to help them look for buses as they cross the street. And pay them well.
@Shubham Rewarding coworkers for finding flaws in their peers’ code sounds like trouble brewing. If the rewards are high enough, the senior devs will rip thru the junior’s code, and the junior guys might not even be able to spot errors in the senior’s stuff. Plus, some “bugs” are ambiguous and intentional. Seems ripe for argument and bad feelings.
Jason
on 11 Oct 11One partial solution is that new hires get assigned to maintenance for the first few weeks. That way, they get introduced to the entire codebase and get a sense of how features are inter-connected. Then, those folks “graduate” to creating new features.
At that point, pair programming and code reviews address some of the “hit by a bus” (or the less bloody “winning the lottery”) issue. The rest is by management being willing to slow down the timeline so that “Pratik” gets his hands dirty.
-jul-
on 11 Oct 11David, I believe this is the wrong approach of pairing. One should switch pairs often. This way knowledge can spread.
Rob P
on 11 Oct 11Im dealing with something exactly like this right now.
I am the territorial one and I have been trying to train and teach a new hire. I first tried to show them how to create a new widget with the framework I created. That was sorta rough and she didn’t quite understand why all of the steps were needed and the complexity that came with it.
There were a couple of new features I wanted to add so instead of trying to teach her the existing steps & logic, we worked closely with thinking about the new features and she was involved and contributed to the overall project. Now she understands the new feature thoroughly and has a deeper understanding of how it works in the world.
Mårten
on 11 Oct 11Great post! typo: ” that it’s going to take Sam” => “than it’s going to take Sam”
Commoditize the infrastructure
on 11 Oct 11The only way to deal with this is to work relentlessly on minimizing the infrastructure. Most of the thorny issues you bring up are related to the underlying complexity wrought by the demanding infrastructure. As soon as we commoditize the infrastructure, the headaches disappear.
A perfect case illustrating that is Rails. Before Rails, we all had to wrestle with the maddeningly complex MVC infrastructure, and there were many ‘swamp guides’ in our development teams who were the only ones qualified to go into the dragon’s lair. Since the introduction of Rails, these things became barely remembered history.
Same is with many other things vying for the developers’ attention today. They all need to be commoditized. Heroku seems to be leading the way these days (check their Add-ons section).
Scott
on 11 Oct 11Convention conquers complexity.
Ben
on 11 Oct 11All organizations have these problems.
What’s funny is that domain master issues are some of the problems that Basecamp and Highrise help solve. Opening up the data so we don’t have to wait for Frank to get back from vacation has helped us a ton.
I know it’s obvious, but good comments go a long way toward explaining the thought process behind the code.
Joe
on 11 Oct 11Comments +1. http://tomdoc.org/
Deltaplan
on 11 Oct 11@Ben : good comments are not enough. Because comments are only useful when you know which part of the code you should be looking at. Which leads to two problems :
- code duplication : even if there is an already well programmed and well documented class for a particular task, if other programmers aren’t aware of its existence they may just redevelop their own class to do the same thing. You may reduce this risk by having at lease one senior developer (possibly the project manager) who keeps an eye on everything that’s developed, and tries to get at least a grasp on the “big picture”. Because ultimately it’s a high-level architecture problem. Another way to deal with it is to put an incentive for developers to clearly identify the code that they have written that is intended to be reused (which means that they are loosing control over it, once it has been reused they can’t just modify it the way they want), and the code that is NOT intended to be reused (which means that they may have more freedom to adjust it to their needs, but they won’t get much credit for it from their peers)
- debugging : when something gets wrong, figuring out the reason why can be really challenging for someone who isn’t familiar with the whole program (ie : any developer…), while the bug would have been seem clearly obvious to another developer (except that he’s not the one who is currently looking at it for some reason). One excellent way to reduce this problem is to ask developers to include well chosen logging messages in their code. Having a clear log of what happened just before the incident gives out clues about which parts of the code to look at, and sometimes it also helps choosing the best developer for solving the problem.
James Adam
on 11 Oct 11If something is hard, the only way to get better is to do it, and do it often. If some of your developers are struggling to work effectively on particular parts of your infrastructure, they should be spending more time working there.
One way for Pratik to get up to speed with “Stacker” is to actually work through the issue along side Sam, rather than give Pratik a shot, and then ask Sam to cleanup afterwards. You can call this pairing if you like, but it doesn’t need to be formal.
Pairing is hard work, and it’s not for everyone, but it’s the best way we’ve found to avoid this problem.
At Free Range, we practice promiscuous pairing (see Rick’s comment above). Not only does it defend against high degrees of territoriality (we’d call them “knowledge silos”) either in individuals or small groups. Even better, it’s also a great way to share knowledge about practices and general approach.
Regardless, Pratik is kick-ass anyway, there’s no code he can’t fix ;-)
Micah
on 11 Oct 11As a manager, and lead architect, my employees are to develop in a fashion in which I prescribe. This allows for creativity in solutions, but also requires discipline to develop a solution that any one of my employees can easily step into.
The only spontaneous learning is concerning the business needs.
If the developers do not like this, then they need to propose why another approach is more appropriate than the development guidelines. If there continues to be discrepancies between manager and employee then the relationship can end without wasting anyone’s time.
If good coding habits are developed for the entire team, then territorial development is not an issue. This is a management issue, not a developer issue.
Anonymous Coward
on 12 Oct 11Glad our shop is not the only one suffering from this syndrome. Often it’s not territoriality, but just the belief that it’s more efficient to have The Original Creators fix and work on sections of code they designed and wrote themselves. In the short-term, it probably is more efficient. In the long run, it becomes less and less so. Then everyone moves on and no one understands anything.
By the way, it’s ‘woven’, not ‘weaved’. (Sorry, not trying to be the grammar police)
Nithin Bekal
on 12 Oct 11It’s strange that I actually got a phone call just before reading this post, saying, “Your program for parsing [some data file] failed today.”
I think it’s bad for teams when people start associating certain parts of comlpex software with individuals, and it gets worse when this kind of behaviour is encouraged in the name of improving efficiency.
Tore Hagen
on 12 Oct 11In my current project we have succeeded with minimizing the code ownership. The key is:
- Make it a priority that everyone works on everything
- Consider who should do what to improve the knowledge distribution during sprint planing
- One code/design review per week where the main goal is not to find bugs, but to consider the design and share knowledge
We are an 8 person team and have been changing out persons with regular interval over the last years, and have always managed that without significant loss of knowledge.The team has 4 different CPUs and 10 different compile targets so it could easily have ended up with a lot of small kings.
Matt
on 12 Oct 11Fun game to play: don’t look at who from 37s wrote the post and try to guess when you’re done. They all, for the most part, have distinctive writing styles and terminology.
This one had me fooled since there wasn’t any foul language :)
Saeed Neamati
on 12 Oct 11While I always think about how to loosely couple developers to projects, so that in critical time any developer can do the rest of the job, I have that experience that letting a developer do his/her job and giving him/her the required autonomy and authority is much more better.
The only problem which remains to be solved is to make your employees become more loyal to you.
Deltaplan
on 12 Oct 11@Saeed
... and more resilient to buses.Anthony Barone
on 13 Oct 11Two thoughts:
1. I believe one standard convention for the readme file(s) should summarize how the original programmer approached solving the problem programatically. When I jump into a piece of code, it is helpful to understand the the big picture and approach the original creator took. I view this differently than the purpose of comments.
2. Perhaps the readme file(s) could be in the form of a post. Other contributors could have an ongoing dialog via comments for questions and clarification.
Sami
on 13 Oct 11The best way to avoid this is well structured and well written code. I don’t think the big issue is how long it takes to implement a feature or fix a bug compared to the original programmer, but that you can’t predict if you’re changes will alter bugs some where else.
So the solution, write great code and great tests! :)
Sami
on 13 Oct 11Update: If you’re happen to get some spaghetti on your hands, I think it’s responsible to say that “I can do this changes, but because of the bad code an lack of tests, it’s hard to predict what bugs can appear else where, so I suggest you pass this on to the original author.”
That’s caring about the customer!
Deltaplan
on 13 Oct 11@Sami : about your comment on spaghetti code, that’s precisely the reason why I encourage my developers to clearly distinguish between code that is meant to be reused, and code that is not meant to be reused.
Because code that is meant not to be reused can be modified much more safely, with less risks of creating unpredictable bugs on unrelated parts of the software. But reused code is meant to stick to the way it was working (which means that above all, the expected behavior of the code should be clearly specified).
So when finding spaghetti inside reused code, then you know that you’re into REAL trouble… but on the other side, reused code is necessarily clearer about the behavior it should have, so as long as it’s not also white-box code, adding proper tests before refactoring will be generally something possible.
This discussion is closed.