Manifesto for publication in Feb 2015. Tom Cully, August 2014
As engineers, many of us in our careers have been unlucky enough to be part of a large failed project. I present the following well known examples:
NovoPay Errors and Issues (Wikipedia)
NPfIT Errors and Issues (Wikipedia)
Healthcare.gov Criticisms (Wikipedia)
Universal Credit Criticisms (Wikipedia)
There are of course thousands other examples both public and less well known. I leave it to the experience of the reader and the industry at large to name and shame - how many times have you worked on a deathmarch or been blamed for a concept failure, and asked yourself, 'How in all the hells did this get so fucked up?'
We are told: a hundred and one reasons. Politics, lack of skilled engineers, bad communication, etc. etc.
I do not accept this. I do not accept that politics is a valid reason even within government systems - even after 17 years I'm still convinced that people just aren't that stupid - futhermore I refuse to accept Sturgeons Law1 when it comes to my colleagues over the years, the vast, vast majority have been competent, professional and intelligent individuals who care about their jobs and the systems they produce.
I propose that the major reason these failures occur, is the attitude and competence of the Architect - the most senior member of the team.
You may say that I'm singling out one role in a complex system unfairly. My question is this: Where does the buck stop? Surely the most senior technical member of the team is responsible for the timeliness and quality of the system produced - someone, in the end, has to be accountable. Architects are also paid more than any one member of a development team, and sometimes more than many members of the management they answer to - such earning potential should reflect the responsibility it respresents - unlike the bankers who can crash a global economy, and still pay themselves millions in bonuses regardless.
The issue from my perspective, broadly, is that IT is all about people, and that some personalities through disposition or choice are poorly equipped to produce what I percieve to be the point of large systems engineering - that is, an operative, performant, available and stable system, with a useful lifespan of at least 10-20 years. Given this, it is reasonable to postulate that the successes of large IT projects are largely down to the more reasonable personality traits of the Architects involved, and that the common failure of large IT projects could well be due to the failings in personality of those same most senior members of our teams, that is, the Architects in question.
In brief, we're definitely getting it wrong somewhere - and, the top technical person in a project holds ultimate responsibility for that, no matter which way you cut it. Evil Architects can and will destroy your project. If the conductor is incompetent, the music will be sour.
We've all worked with or under these people. If the reader will excuse my whimsy, I'll characterise them using the AD&D alignment system, that is, two axis, Good-Neutral-Evil and Chaotic-Neutral-Lawful. These caricatures may seem over the top in the way I'm describing them - I am taking some poetic license here for the purposes of illustration - but I think the reader may agree; I can think of a specific example of each, recorded here with little by way of embellishment. This is a rant, sure - but it's a rant based on real experience.
Lawful Evil. The BPB - at least in their own mind - is an irrefutable genius, most likely with a long history of projects, but has no connection with - or for that matter much interest in - the engineers and testers who will actually implement The Grand Vision™. BPBs commit the cardinal sin of the highly intelligent - the assumption that no-one else is more intelligent, or for that matter even close - and also the cardinal sin of the highly experienced - the assumption that no-one else's experience is remotely as valid or as in-depth. This belies a hierarchical and linear way of seeing the world; Stupid < Mundane < Intelligent and Naive < Competent < Experienced; a simplistic world view that ignores the fact that one may well be a genius in e.g. API design but useless at security, or be able to design complex state machines in one's head but have difficulty with basic scalability concerns.
BPBs tend to reduce every other individual on the project to a glorified typist, only dimly aware of just enough to implement their small part of The Grand Vision™. BPBs are prone to micro-management and over-specification, and when found in large projects usually possess cult-leader charisma, the sort of person who can walk into a room with 20 engineers and testers and argue blind for 2 hours with every single one of them - but each will leave the meeting, still complaining about the obvious insanity of the design, And Sit Down And Do What The BPB Said, Anyway. The BPB is the only Evil Architect archetype that will actually see a deathmarch project through to its inevitable catastrophic failure, although like the BSD and The Bullshitter, you can guarantee they'll make sure the blame lies elsewhere.
BPBs differ from Process Gods and Charismatic Dictators in the fact that they're absolutely, resolutely, sure that they're right at all times and in all regards, and have no interest or respect for anyone else's opinion, even if what they are doing is universally agreed to be insane, and will ignore criticisim, advice and ideas even well past the obvious failure of their architecture.
Neutral Evil. The BSD is much, much more interested in being 'The Man' and developing 'The Man's Career' than any other outcome, regardless of cost. A Machiavellian genius and born corporate climber, the BSD is nonetheless usually technically competent and may well be perfectly capable of an architecture which is both rational and effective. Unfortunately, the BSD has Absolutely No Interest Whatsoever in the Project, the people involved, the stakeholders, or for that matter, the final quality of the system. The BSD isn't even usually especially interested in completing the system - especially if the BSD is a contractor - the project and its staff serve only to bolster the BSD's position in the company, their overall career, their earning potential and/or political power. BSDs are prone to bullying and other dominance plays and - like the BPB - micro-management. Like The Bullshitter, they'll also eject post-haste if they detect for one second that the project outcome may not serve their goals - but not before making sure someone else will drop neatly in the shit for the resulting disaster.
BSDs differ from Charismatic Dictators in that they truly don't give a suffering fuck about the project or the people in it. Both archetypes are interested in power, but the Charismatic Dictator wields it for the good of the project and the people in it, whereas the BSDs crave power and advancement purely for themselves and themselves0 alone.
Incidentally, as with all these dramatic examples, there's no reason why a BSD can't be female, although I'm unsure as to what the appropriate caricature would be. This personality lends itself better to a male description, because it's more socially acceptable for men to act like aggressive arseholes in a corporate environment. The point here is that a BSD is a corporate psychopath in an Architecture role, they can - and do - come in all genders.
Chaotic Evil. The Bullshitter has accepted long ago that they possess neither the technical ability for architecture, nor the interest or discipline to acquire it - but can compensate with a flair for grift, performance and a natural talent for leadership. The key point here is that The Bullshitter is completely self-aware in this regard - and, at least internally, under no illusions as to their own incompetence. Bullshitters are usually highly intelligent regardless - wordsmiths, consummate liars, well versed in corporate buzzwords and doublespeak, and have likely taken the skill of brown-nosing superiors and exec to the level of an artform.
Bullshitters are capable of writing, speaking and presenting in a very motivational manner; they're normally well liked by the project staff - at least initially; but this is only camouflage to hide the fact that The Bullshitter, Fundamentally, Has No Idea What The Fuck They're Doing. This is different from 'Impostor Syndrome', where otherwise capable people suffer a lack of confidence in their role - in short, The Bullshitter knows that it doesn't matter whether they are capable of performing the role they have slimed their way into - what matters is the appearance of performing the role. They know they can't fool all the people all the time, but they also know that the time is limited, and they are quite capable of fooling everyone for just long enough. The Bullshitter is the one Evil archetype who is most likely to have all the paper qualifications for Architect, but to never have actually written a single line of code in a production system in their whole life.
Bullshitters tend to lean heavily on the experience of the team to survive, leading to long, drawn out project failure in the event that the team should be inexperienced, as the Bullshitter finds ever more inventive ways to hide the fact that the project is a complete disaster from those above; or, The Bullshitter will spin their total lack of coordination and direction as 'collaborative process' to exec and stakeholders ad nauseaum, until an experienced team simply self-organises and SkunkWorks the whole architecture and implementation themselves. In this - the only positive outcome for an endeavour headed by such a creature - The Bullshitter will inevitably take all of the credit and move on to larger or more lucrative projects, using the 'success' as further proof of their own excellence - or, if the project fails miserably, they'll have left early, and/or redirected all of the blame onto others well before the ship goes down. Sometimes, they can even exit whilst still enjoying the warm feeling and best wishes of the people they've just screwed.
Of course, Evil is the worst possible case. There are a large number of competent software architects in the world, mirrors to my slightly dramatic examples above. Each has their pros, unfortunately each also has their cons, as detailed:
Lawful Good. Process Gods are competent and intelligent, and usually have a healthy respect for the experience and intelligence of the members of the engineering and testing team. The will insist on extremely high quality of code with a strong emphasis on formal process, testing and peer review, and are often of the opinion that late and working is better than on time and faulty. They will ensure, personally, that every T is crossed and every i dotted, and their tolerance for incompetence and politics is very low.
Unfortunately, Process Gods are prone to Creationism and process micro-management, not to mention perfectionism. Although they respect the experience and intelligence of the team they will tend to insist on a level of formal process that makes even the most simple change to the architecture the engineering equivalent of root canal surgery - necessary, but painful in the extreme.
Neutral Good. Charismatic Dictators as usually both highly competent problem solvers and excellent researchers, they will motivate and inspire their teams and prove dilligent communicators with shareholders and engineers alike. Charismatic Dictators are often vitriolic defenders of their engineers from outside influences and pressures, and will tend to cultivate a big-brother/parent-child relationship with most of those that are, from their point of view, in their care and need their protection. Charismatic Dictators are also usually compulsive teachers, and will actively mentor anyone they see as a weaker member of the team in terms of experience or knowledge.
The cons of such a beast being in charge are that the Charismatic Dictator will expect and demand total loyalty and submission, and may severely punish anyone who is preceived by them to be exceeding their brief or 'acting out'. New ideas and criticisim will be welcome only as long as the supplicant has observed the correct forms of respect due to one of the Charismatic Dictator's station. Their tolerance for mavericks tends to be low even if the ideas they bring to the table are truly revolutionary, and they can be slow to consider or adopt any idea, approach or technology they are not directly familiar with, or hasn't been given Their Stamp Of Approval. Although effective communicators, they are sometimes accused - often for no small reason - of being smug and patronising, and this can lead to friction within the team.
Chaotic Good. The Maverick is a genius anarchist. Mavericks have usually evolved from highly effective engineers or other IT professionals. They have an irreverent style and will usually be actively involved in culture committees and other out-of-office endeavours, and will tend to start such groupings if they don't already exist. As natural leaders, they will tend to motivate and inspire their team in a very 'walk amongst your troops' and 'lead from the front' manner. This, combined with the Maverick's natural charisma and infectious exuberance, is normally well received, as it is a lot easier to respect someone who has done your job in the past and has come up through the ranks - especially when that person is still willing to stand in the trenches with you.
Mavericks tend to advocate a very liberal form of Agile, and will quickly become less effective if subjected to Waterfall or, for that matter, any form of more-than-marginally constrictive process model. They tend to be very open to new ideas and approaches - especially if the idea involves a cool new toy, 'sticks it to the man', or is related to one of their innumerable personal projects. Like Charismatic Dictators, Mavericks will often be very protective of their team from outside pressures, especially the more junior members. They're more likely to hold meetings in a bar than a boardroom, and will tend to strongly blur the lines between their professional and social lives. If there isn't an operative SkunkWorks department in your organisation, your Maverick will start one - on their own, if necessary.
Downsides to the Maverick persona is that as stated, this person will usually have been, in the past, an extremely competent and/or possibly world class engineer, and May Secretly Want To Write The Whole Bloody Thing Themselves. And, they may indeed be capable of doing so - but their natural exuberance and tendency to see the world in larger strokes - leaving the details to their more Lawful spectra bethern - may cause the overall quality of the system to suffer, and they may with the best of intentions ignore the experience of their colleagues, simply out of excitement to get on with things. At worst, this can manifest as unintentional micromanagement of their team, when the Maverick is sure that they are right and excited to Just Get On With It.
In addition, although they are very good at handling change, they may get excited about a new part of the system or a new technology and completely forget about finishing a previous project or component - in the rare worst case, this can lead to fragmentation of the architecture over time, requiring the team to jump through increasingly more complex hoops to keep the whole system running. Mavericks also both despise and are notoriously bad at confirming to formal process, and may ignore - or even actively subvert - existing processes seemingly just for the hell of it.
The overall problem, whether your architect be Good or Evil, is that the success or failure of the project largely comes down to the competence and personality of Just One Person. This is at the least arbitary, and I won't even start on the widespread occurence and consequences of The Peter Principle.
I'm not attacking the role of Architect - I'm attacking the the way the role is defined, in the fact that it allows incompetents to thrive and survive, massively amplifies the mistakes of even those otherwise well suited for the task, and disconnects and ignores the total experience of the team.
I propose, in short, that we remove the single point of failure. In the same way that the DevOps revolution has done much to improve developer understanding of ops, and reduced the load on overworked sysadmins who were used to spending 90% of their time cleaning up other people's mess, and spread the knowledge and responsibility of How Stuff Actually Gets Done over entire teams as opposed to one stressed and hairless individual - I propose a new class of IT Professional, the Developer Architect, or DevArch for short.
Essentially, I propose the end of the concept of Architect being a single role divorced from implementation.
Architecture should ALWAYS be a collaborative effort between a group of senior engineers who are actually working on the system..
A DevArch is a Developer Architect - that is, a senior engineer who works not only in the design and implementation of software, but also forms part of a collaborative group to design, maintain and evolve the architecture of the system on which he or she works.
The first thing to do is re-task your existing architect to be the 'curator' of the architecture. The curator should be capable of producing prototypes and technology demonstrators off their own bat - if they can't/won't code, get rid of them. This is intentionally harsh: people who have no understanding of software implementation - or whose knowledge is not current from a coalface point of view - have no business designing software.
The new curator should form an AWG (Architecture Working Group) - a periodic round table meeting were proposals can be made, prototypes demoed, and consensus can be reached. The purpose of the curator over and above the other members of the AWG is primarily to chair this meeting, coordinate the agenda and to produce and file the meeting minutes. Additionally, when factions develop - as they inevitably do - the curator may - in agreement with management/exec - make a call as to which approach best represents the interests of the project and the organisation.
The curator is, of course, free to propose changes to the architecture, just like anyone else. It's best that you put some semi-formal procedure around proposals for changes, and manage the architecture documentation in a public accessible system (GitHub Wiki, Atlassian Confluence, MediaWiki, or the like).