Let me warn you that this particular post will be very, very opinionated and equally direct. I share many of the same opinions with mid-level leadership and engineers at mass, the folks that are in the true trenches with respect to outsourcing consequences and reality. If you have an opposing viewpoint I doubt we will share common ground on any of the topic. I feel strongly enough on the subject that I left a previous employment before an outsourcing quicksand project drew me into the quagmire and have not regretted the moment even a little bit. I have witnessed first-hand the consequences of numerous outsourced projects and I'll speak to why I will foreverly believe them to be a project-killing, corporate chaos-bringing idea.
One last thing before we get started, this post is not meant to discredit the capabilities of engineering teams in the Eastern Hemisphere or elsewhere. I've had the privilege of working with extraordinary engineers from all over the globe and my criticisms do not lie with the engineering teams, rather they revolve more about logistics and expected 'cost savings'.
Lets get our feet wet;
It always begins the same, some C-suite suit who clearly doesn't understand any form of engineering gets a brilliant idea. It comes in the form of "we should outsource to the other side of the planet".
Why would they do this you may ask? The reasons will come in many forms, 'to enhance quality of service', 'access to worldwide intellectual capital', 'because of business needs',... If you've ever been on the receiving end of any such rationale and your bullshit meter trips 11 don't question the meter readings. Business speak invented these terms to mask the real motivation. The reality is always, always, always cost savings....or at least the perceived expected cost savings.
But shouldn't we be conscience of our spending and look into all aspects of expenditures? Yep, absolutely, but you'll find a dichotomy in executive thought when it comes to what departments should be outsourced. For example, I'm confident that you could fully replace a chief executive (CEO, CFO, CIO,....) with less expensive personnel from Asia, India...but that would be ludicrous! Sure, ok...how 'bout just one? Pick a chairperson and replace them with an Oxford-trained executive in London and you'll be faced with the same consequences that outsourcing engineering encounters on a daily basis. Chew on that for a moment before moving on.
Timezones, Timezones, Timezones
The secondary villain, besides the mindless Neiman Marcus suit hanger that made the oursourcing decision, is the dreaded timezone. Executive choose not to understand how effective engineering teams operate and fall under the illusion that programmers prefer working the 3 am shift. How else would the idea of spreading a team across the planet would make sense? With vacation homes across the globe, executives certainly understand the concept of timezones, but choose not to recognize its consequence on team productivity.Let's say all teams work 9-5 local time, pretty standard worldwide as far as I can tell.
- Chicago and London share a whopping 2 hours of shared time as a result. Got a question with a system component in London, find a common block of time in that 2 hrs. Miss your window? Wait a day.
- Chicago and India share 0 hours of shared time. That means a 24 hour delay for every team interaction via e-mail.
- Chicago to Romania share 0 hours as well.
Each of these countries have extraordinary university engineering programs and matching talent. The shared nemesis is timezones.
So how is it that executives tout the need for team communications while banging on the outsourcing drum? The expectation is simple; the teams should simply adjust their schedule. I've worked for numerous managers that worked 4am to 6pm daily simply to account for leading local and remote teams. You know how long they lasted? Milk has been known to spoil faster than some of those assignments. Easy to fix, right? Just make the teams adjust their schedule. Who doesn't love the 9pm shift; EVERYONE, that's who! Nobody likes night shifts, only desperate folks are willing to undergo such an assignment. "Why can't we retain talent?", "Why are you behind projected schedule?", "What is the delay with the Bucharest team?" Unfortunately few have the balls or opportunity to simply say "Because you made the hair-brained decision to make it nearly impossible to maintain an effective team!!"
True leaders lead by example, chief executives that belittle the impact of such remote teams should work the graveyard shift or ship one of your executive staff (the one's you collaborate routinely with) to London and continue to run the company for a year or so. Schedule a series of board meetings for 3 am and see how effective that works. Alternatively, you can recognize that companies typically co-locate their executive staff to enable collaboration and effective execution of a company.
Hidden Costs
Engineering budgets are pretty heavy-weight, making up a substantial amount of corporate expenses. At $60-80 / hour salaries they can easily be a prime target for reductions. You know what's more expensive than an engineers hour of time? An unproductive hour of time of engineering time. If you hate dropping $60-80 at your next hot lather shave, think how sore you'll be dropping the cash and not getting the service. Your role should be enabling the most productive teams to develop the suite of products that will make your company legendary rather than making them less productive and less effective.But hey, we're engineers and we thrive to inform, measure and open to new ideas. We love to gather data and I've seen the same idea presented from numerous teams "enable us to measure the cost of bad ideas", let us track and report loss of productivity, measure communication latencies, identify and measure the consequence of 'blockers'. Data-driven business decisions, right? Allocate a charge number to track unproductive hours and blocked tasks. Alas, while that would allow a data-driven decision and improvement process that idea is struck down with vengeance at every company I've worked for. Head-in-sand executives would rather pretend like its a good idea than recognize that they made a foolish decision.
Illusion of Cheaper Development Costs
The path to outsourcing always begins with the call of the sirens, the seductive and enchanting calls meant to draw a corporation onto the rocks. The separation of myth from reality take critical thinking, asking the right questions and challenging the answers and the executive elite are typically don't ask such questions. Additionally, they tend to make the decision in a vacuum of real stakeholders. But to be fair, who is better to decide the best process for creating effective engineering teams than MBAs, salespeople, and board members and 'thought leaders'.
Let me tell a tale of how it very well could go;
Some salesperson chats with another salesperson and is informed that Romania software developers are readily available at the budget price of $20-22/hr, "why aren't we taking advantage of this?" Certainly these two folks have somehow stumbled upon the greatest silver bullet of cheap and effective software product development!! One salesperson bends the ear of the vice president of engineering, a board member and the journey begins.
A series of telephone conversations begin between the company and overseas outsourcing firms, while many red flags present themselves during the calls it becomes clear that the decision was already made. Travel is arranged for face-to-face meetings in Romania for more detailed conversations and review of the facilities. Surprisingly (to some) Romanian software consulting firms face the same challenges those in the US; underpaid engineers are quick to leave for more lucrative salaries, entry level engineers lack experience and knowledge of leading edge technologies are hard to come by. A revolving door of ill-experienced workers coming in the door, gaining a few months of experience which makes them marketable, then back out the door for more lucrative employment. The $22/hr experience software engineer is a myth. Inexperienced code-camp exports and internships can be readily employed right here as well. The year, 2016, the required tech C#/WPF, AngularJS, MySql, SqlServer; how many $22/hr Romanian developers fitting the bill - exactly 0, but countless readily available to learn the tech on the clock (same as in the US). More seasoned consulting firms with veteran develops had the skills at $90/hr....unsurprisingly about the same rate as those in the US.
Rookies are cheaper than senior engineers, universally.
India, Romania, China,....engineers are aware of their market worth and will quickly acquire it.
You get what you pay for, simple as that; Romanian Leu, Rupees, Euros or dollars.
How does the journey end? The engineering manager that would be responsible for this dumpster fire recognizes it as an inevitability and fled that wildfire like a squirrel with it's tail on fire.
Let me tell a tale of how it very well could go;
Some salesperson chats with another salesperson and is informed that Romania software developers are readily available at the budget price of $20-22/hr, "why aren't we taking advantage of this?" Certainly these two folks have somehow stumbled upon the greatest silver bullet of cheap and effective software product development!! One salesperson bends the ear of the vice president of engineering, a board member and the journey begins.
A series of telephone conversations begin between the company and overseas outsourcing firms, while many red flags present themselves during the calls it becomes clear that the decision was already made. Travel is arranged for face-to-face meetings in Romania for more detailed conversations and review of the facilities. Surprisingly (to some) Romanian software consulting firms face the same challenges those in the US; underpaid engineers are quick to leave for more lucrative salaries, entry level engineers lack experience and knowledge of leading edge technologies are hard to come by. A revolving door of ill-experienced workers coming in the door, gaining a few months of experience which makes them marketable, then back out the door for more lucrative employment. The $22/hr experience software engineer is a myth. Inexperienced code-camp exports and internships can be readily employed right here as well. The year, 2016, the required tech C#/WPF, AngularJS, MySql, SqlServer; how many $22/hr Romanian developers fitting the bill - exactly 0, but countless readily available to learn the tech on the clock (same as in the US). More seasoned consulting firms with veteran develops had the skills at $90/hr....unsurprisingly about the same rate as those in the US.
Rookies are cheaper than senior engineers, universally.
India, Romania, China,....engineers are aware of their market worth and will quickly acquire it.
You get what you pay for, simple as that; Romanian Leu, Rupees, Euros or dollars.
How does the journey end? The engineering manager that would be responsible for this dumpster fire recognizes it as an inevitability and fled that wildfire like a squirrel with it's tail on fire.
Retain Our Core Business
Let me finish with another fallacy that can masquerade as a reasonable idea; "we will retain our core business and outsource just the front-end".Let me rephrase that for you so you can recognize the flaw of that idea;
'We will outsource the customer-facing interface, the one that makes the first impression to our users to a non-native speaking consulting firm who may/may not be aware of the cultural and social expectations of the product'. Surprisingly to no-one, travel systems, financial products, and business infrastructure, and nearly every other domains have subtle distinctions in other countries. Like the infamous Spaghetti Westerns, your product may become forever memorable as a quirky and slightly off product rather than the quality product you're shooting for.
That's it, my very blunt opinion on outsourcing co-development across the globe. 1000% wrong decision and unfortunately a one-way track because ego-fueled executives will never acknowledge their mistake, saving face comes at the sacrifice of everything else. But hey, as my lovely wife has repeatedly said; "I make a very lucrative living fixing outsourced projects".
If you choose to outsource, outsource the entire project, decision-makers and all. Don't fool yourself that you can make collaboration work or that your team will permanently adjust their lives to make it work. The world is rich with intellectual talent and smart companies will take advantage of them, dumb companies will do it poorly. Real leaders will make mistakes but will take ownership of the decision, perform retrospectives on the decision and modify it as needed and apply it to future decisions.
The flaw in your "there really is no cost reduction" argument is that the corporate bean counters are too dumb to notice it. You may not get them around to understanding how software development works, but surely they do understand money?
ReplyDeleteSpeaking of software development, it is *mostly* the "low" end of the work that gets outsourced to destinations like India: you get cheap code monkeys and test monkeys to fix bugs and script tests at one-fourth to one-third of what it takes you to get the same thing done in your country. In the first place, you'll have a hard time finding engineers who are willing to do that unsexy piece of work: because engineers with the needed skills (which are only midlling-level by the way, you don't need super-sharp guys to do the outsourced kind of work) are in short supply in your market, and because those that are available have creative ambitions.
There are great engineers in India too, as you acknowledge, but they prefer to emigrate and work abroad, not just for money, but also because their own creative ambitions are not fulfilled in outsource sweat shops back home.
My opinion is that the high end of the work too can be done in a geographically distributed fashion. The limitation here is not really the time-zone, but technical imagination. If the product is architected to be highly modular with lean interfaces, then the modules can be developed independently of each other. This can be done. Someone said that software product architectures tend to reflect the structure of the organisations that develop them. Ownership, turf and territoriality, you know. If so, why can't the product be architected from the ground up factoring outsourcing in? Because you don't really like the outsource shop to own anything!
(While on topic, life of managers on this side of the globe is not great either; worse in fact. I was a manager once, working for India offices of Bay Area companies. It is generally the HQ that calls the shots, so meeting schedules are set to suit the convenience of managers out there).
By the way, I chanced on your blog googling for ffmpeg-related info. Great info, thanks.