Sunday, April 26, 2020

Eastern Hemisphere Outsourcing and Other Horrible Ideas


Let me warn you that this particular post will be very, very opinionated and equally direct.  I share many of the same opinions with mid-level leadership and engineers at mass, the folks that are in the true trenches with respect to outsourcing consequences and reality.  If you have an opposing viewpoint I doubt we will share common ground on any of the topic.  I feel strongly enough on the subject that I left a previous employment before an outsourcing quicksand project drew me into the quagmire and have not regretted the moment even a little bit.  I have witnessed first-hand the consequences of numerous outsourced projects and I'll speak to why I will foreverly believe them to be a project-killing, corporate chaos-bringing idea.

One last thing before we get started, this post is not meant to discredit the capabilities of engineering teams in the Eastern Hemisphere or elsewhere.  I've had the privilege of working with extraordinary engineers from all over the globe and my criticisms do not lie with the engineering teams, rather they revolve more about logistics and expected 'cost savings'.

Lets get our feet wet;

It always begins the same, some C-suite suit who clearly doesn't understand any form of engineering gets a brilliant idea.  It comes in the form of "we should outsource to the other side of the planet".



Why would they do this you may ask?  The reasons will come in many forms, 'to enhance quality of service', 'access to worldwide intellectual capital', 'because of business needs',... If you've ever been on the receiving end of any such rationale and your bullshit meter trips 11 don't question the meter readings.  Business speak invented these terms to mask the real motivation.  The reality is always, always, always cost savings....or at least the perceived expected cost savings.

But shouldn't we be conscience of our spending and look into all aspects of expenditures?  Yep, absolutely, but you'll find a dichotomy in executive thought when it comes to what departments should be outsourced.  For example, I'm confident that you could fully replace a chief executive (CEO, CFO, CIO,....) with less expensive personnel from Asia, India...but that would be ludicrous!  Sure, ok...how 'bout just one?  Pick a chairperson and replace them with an Oxford-trained executive in London and you'll be faced with the same consequences that outsourcing engineering encounters on a daily basis.  Chew on that for a moment before moving on.

Timezones, Timezones, Timezones

The secondary villain, besides the mindless Neiman Marcus suit hanger that made the oursourcing decision, is the dreaded timezone.  Executive choose not to understand how effective engineering teams operate and fall under the illusion that programmers prefer working the 3 am shift.  How else would the idea of spreading a team across the planet would make sense?  With vacation homes across the globe, executives certainly understand the concept of timezones, but choose not to recognize its consequence on team productivity.

Let's say all teams work 9-5 local time, pretty standard worldwide as far as I can tell.

- Chicago and London share a whopping 2 hours of shared time as a result.  Got a question with a system component in London, find a common block of time in that 2 hrs.  Miss your window?  Wait a day.
- Chicago and India share 0 hours of shared time.  That means a 24 hour delay for every team interaction via e-mail.
- Chicago to Romania share 0 hours as well.

Each of these countries have extraordinary university engineering programs and matching talent.  The shared nemesis is timezones.

So how is it that executives tout the need for team communications while banging on the outsourcing drum?  The expectation is simple; the teams should simply adjust their schedule.  I've worked for numerous managers that worked 4am to 6pm daily simply to account for leading local and remote teams.  You know how long they lasted?  Milk has been known to spoil faster than some of those assignments.  Easy to fix, right?  Just make the teams adjust their schedule.  Who doesn't love the 9pm shift; EVERYONE, that's who!  Nobody likes night shifts, only desperate folks are willing to undergo such an assignment.  "Why can't we retain talent?", "Why are you behind projected schedule?", "What is the delay with the Bucharest team?" Unfortunately few have the balls or opportunity to simply say "Because you made the hair-brained decision to make it nearly impossible to maintain an effective team!!"

True leaders lead by example, chief executives that belittle the impact of such remote teams should work the graveyard shift or ship one of your executive staff (the one's you collaborate routinely with) to London and continue to run the company for a year or so.  Schedule a series of board meetings for 3 am and see how effective that works.  Alternatively, you can recognize that companies typically co-locate their executive staff to enable collaboration and effective execution of a company.

Hidden Costs

Engineering budgets are pretty heavy-weight, making up a substantial amount of corporate expenses.  At $60-80 / hour salaries they can easily be a prime target for reductions.  You know what's more expensive than an engineers hour of time?  An unproductive hour of time of engineering time.  If you hate dropping $60-80 at your next hot lather shave, think how sore you'll be dropping the cash and not getting the service.  Your role should be enabling the most productive teams to develop the suite of products that will make your company legendary rather than making them less productive and less effective.
But hey, we're engineers and we thrive to inform, measure and open to new ideas.  We love to gather data and I've seen the same idea presented from numerous teams "enable us to measure the cost of bad ideas", let us track and report loss of productivity, measure communication latencies, identify and measure the consequence of 'blockers'.  Data-driven business decisions, right?  Allocate a charge number to track unproductive hours and blocked tasks.  Alas, while that would allow a data-driven decision and improvement process that idea is struck down with vengeance at every company I've worked for.  Head-in-sand executives would rather pretend like its a good idea than recognize that they made a foolish decision. 

Illusion of Cheaper Development Costs 

The path to outsourcing always begins with the call of the sirens, the seductive and enchanting calls meant to draw a corporation onto the rocks.  The separation of myth from reality take critical thinking, asking the right questions and challenging the answers and the executive elite are typically don't ask such questions.  Additionally, they tend to make the decision in a vacuum of real stakeholders.  But to be fair, who is better to decide the best process for creating effective engineering teams than MBAs, salespeople, and board members and 'thought leaders'.

Let me tell a tale of how it very well could go;
Some salesperson chats with another salesperson and is informed that Romania software developers are readily available at  the budget price of $20-22/hr, "why aren't we taking advantage of this?"  Certainly these two folks have somehow stumbled upon the greatest silver bullet of cheap and effective software product development!!  One salesperson bends the ear of the vice president of engineering, a board member and the journey begins.
A series of telephone conversations begin between the company and overseas outsourcing firms, while many red flags present themselves during the calls it becomes clear that the decision was already made.  Travel is arranged for face-to-face meetings in Romania for more detailed conversations and review of the facilities.  Surprisingly (to some) Romanian software consulting firms face the same challenges those in the US; underpaid engineers are quick to leave for more lucrative salaries, entry level engineers lack experience and knowledge of leading edge technologies are hard to come by.  A revolving door of ill-experienced workers coming in the door, gaining a few months of experience which makes them marketable, then back out the door for more lucrative employment.  The $22/hr experience software engineer is a myth.  Inexperienced code-camp exports and internships can be readily employed right here as well.   The year, 2016, the required tech C#/WPF, AngularJS, MySql, SqlServer; how many $22/hr Romanian developers fitting the bill - exactly 0, but countless readily available to learn the tech on the clock (same as in the US).  More seasoned consulting firms with veteran develops had the skills at $90/hr....unsurprisingly about the same rate as those in the US. 
Rookies are cheaper than senior engineers, universally.
India, Romania, China,....engineers are aware of their market worth and will quickly acquire it.
You get what you pay for, simple as that; Romanian Leu, Rupees, Euros or dollars.
How does the journey end?  The engineering manager that would be responsible for this dumpster fire recognizes it as an inevitability and fled that wildfire like a squirrel with it's tail on fire.
  

Retain Our Core Business

Let me finish with another fallacy that can masquerade as a reasonable idea; "we will retain our core business and outsource just the front-end".

Let me rephrase that for you so you can recognize the flaw of that idea;
'We will outsource the customer-facing interface, the one that makes the first impression to our users to a non-native speaking consulting firm who may/may not be aware of the cultural and social expectations of the product'.  Surprisingly to no-one, travel systems, financial products, and business infrastructure, and nearly every other domains have subtle distinctions in other countries.  Like the infamous Spaghetti Westerns, your product may become forever memorable as a quirky and slightly off product rather than the quality product you're shooting for.



That's it, my very blunt opinion on outsourcing co-development across the globe.  1000% wrong decision and unfortunately a one-way track because ego-fueled executives will never acknowledge their mistake, saving face comes at the sacrifice of everything else.  But hey, as my lovely wife has repeatedly said; "I make a very lucrative living fixing outsourced projects".

If you choose to outsource, outsource the entire project, decision-makers and all.  Don't fool yourself that you can make collaboration work or that your team will permanently adjust their lives to make it work.  The world is rich with intellectual talent and smart companies will take advantage of them, dumb companies will do it poorly.  Real leaders will make mistakes but will take ownership of the decision, perform retrospectives on the decision and modify it as needed and apply it to future decisions.





Sunday, April 19, 2020

Mobile Development Is All Edge Cases


I've been a professional software developer since '98, that's 22 years of doing this stuff.  Most of my career has been in embedded development, but the field is pretty broad, I'm not a guy that's coding on 8-bit baby processors but rather fully capable 32-bit Posix-based operating systems, a couple of gigs of memory and peripherals.  Even so, mobile development (that on phones, tablets, or IOT) is arguably more complicated.

The complexity can be misleading and under-appreciated, partly because the misconception "It's just Java, you know Java?", partly because the Android libraries manage a good deal of the complexity, and partly because many true complexities are simply overlooked.  Let's just peek at the complexities of mobile development and hopefully you'll leave with a new perspective.

Multi-Applications

Unlike most embedded devices, phones/tablets have a never-ending list of applications that can be installed, run, uninstalled and such.  The user may choose to launch one, many or all applications at any given time.  Typical embedded devices have a fixed installation, dedicated purpose and go through a heavy quality assurance procedure.

Resource Reclaimation

Mobile applications are contending for device resources with all other applications.  The applications are independent and isolated from one another, but one complication that impacts each app is the possibility of the operating system terminating or restarting your app to preserve the resources for a contending app.  In other words, a properly developed app should gracefully address it being terminated at any time in order to reclaim the apps resources and give them to another.  This is the whole purpose of the Activity States, the appropriate handlers and a proper app should consider the impacts of it being terminated unexpectedly at any time, doing anything.

Network Connectivity

Dealing with wireless communication itself is a extremely complicated problem in itself.  A common misconception is that "with 4G connectivity is everywhere".  Sure, but unlike a fixed-line network connection, wireless signals come and go, especially when you are in motion.  Additionally, the type of network connection may change, yet still have a network connection.  This can manifest itself as a brief, or perhaps not brief, disruption of service.  The app could hang a bit, or worse simply crash.  If you want to observe an example of this, go into your garage, connect to your home network, start playing YouTube, then drive away.  The app will need to manage a network connectivity exchange from wifi to 4G, hopefully seamlessly; but don't count on it.  Managing this in the process of a network exchange can be a bugger.

Abrupt Termination

The operating system can clobber your app to reclaim resources, as can the user.  At any moment, regardless of what your app is currently doing, the user can simply pull-the-plug.


It's all edge cases, you simply can't rely on anything for a robust application.  You can ignore them, and your app can not manage them properly and honestly....you may get away with it.  Simply ignoring them however doesn't make them go away and if you're looking to develop a 'perfect app', these are the types of complications you need to consider and address.  You're standing on a stool and at any opportunity the user, operating system, resources can be abruptly kicked out from under you.  Handle every kick, or be prepared for a fall.

Sunday, April 12, 2020

The Extinction of Mentorship



Perhaps 'extinction' is a bit strong, but I certainly feel it's on the endangered species list.  I have a sincere hope that I'm way off and mentorship is alive, well and booming.

When I was a wee lad, growing up in fly-over land USA, my father was building and finishing our family  house.  My older brother was extraordinarily skilled in many things, sheetrocking would be one of them.  He and my father would hang sheetrock and finish it with a proficiency that truly amazed as we younger kids looked on.  My father, in time, took pity on me as it was clear I wanted to try and he handed the sheetrocking screw gun.  Properly sunk screwheads were replaced with over-sunk, under-sunk and cockeyed counterparts.  Clearly, this rookie was adding work to a proficient, efficient team.
The entire 2200 sq foot house required sheetrocking, weeks of work I continued to assist, gaining proficiency along the way.  To this day I'm convinced that my participation only slowed the entire process.  The goal of apprenticeship is to exchange skills with the overall goal of eventually having a larger skilled staff.  To be a financial win this means recouping the training costs with improved efficiency of the new staff.  It's playing the long game, investing in your team with expected future gains.

Certainly there are other goals of mentorship, apprenticeship or training journeymen, but if you focus on it purely financially there are new challenges to the current world than that of the years past.  Loyalty to company and loyalty to employees have taken a big hit these past decades so investing in apprenticeships has risk of net-loss.  Like my sheetrocking activity, the progress of our house was worsened by my participation; simply couldn't recoup the cost over a few weeks.

Active mentoring an apprentice takes time, likely a lot of time.  I'm not talking an weekly meetup with a mentee with some inspirational quotes of a bit of advice.  I'm talking daily involved monitoring and guiding an engineer in all aspects of the field.

I'm the engineer I am today because I was lucky enough to have a mentor early in my career, my cube neighbor and boss.  Sadly, I've not witnessed real mentorship since, 14 years and counting, because all the teams I've been part of since have been senior-leveled experience.

So, where do young engineers go for advice?  Reddit, slack, Twitter,...?

I'm looking for reassurance that mentorship is still alive and well.

Cheers.

Sunday, April 5, 2020

Allen Newell -- Desires and Diversions

When I was in graduate school I stumbled upon a batch of VHS lecture series about Computer Science, most 'dry as dirt', but this particular video I found interesting and in-fact inspirational.  I've watched it more times than any entertainment DVD in my collection.  Every once in a while, when I'm struggling to remember what I love about the discipline, this video reboots my inspiration;


I hope you enjoy it as much as I do.