If automation were as flawless as some vendors make it out to be, unemployment would be a lot higher. Sure, automation can save time, money, and reduce human-induced errors. Machines don’t need coffee breaks, lunch breaks, or vacations. On the other hand, automated errors happen much faster than manual errors, sometimes yielding expensive or embarrassing results – often because something important was overlooked.
Hardware and Software Glitches Rock Major Stock Exchanges
Take Knight Capital, for example. A software-related error caused a series of erroneous NYSE-listed securities orders. Apparently the trades were supposed have taken place over the course of several weeks. The result was a pre-tax loss of $440 million in 20 minutes.
The company’s stock plummeted 75%. Subsequently, Knight sought and received $400 million in exchange for convertible shares that diluted shareholder positions. A class action suit was recently filed on behalf of shareholders in the United States District Court for the Western District of Tennessee citing the software error as a breach of duty.
A software error was also to blame for Facebook’s 30-minute IPO delay. That led to a series of lawsuits and speculation that the error had deflated Facebook’s value. Nasdaq set aside $62 million to compensate brokers for their losses, although collectively Knight Capital and others supposedly lost $500 million.
And, the Toyko Stock Exchange halted trading for 4.5 hours in February 2012 due to a server error. Apparently, the automatic backups also failed. Six months later, a networking glitch halted derivatives trading at the Toyko Stock Exchange for more than 90 minutes. Once again, automatic backups failed. While the extent of the fallout has been swept under the rug, the CEO, CIO, and other IT executives were subjected to pay cuts.
In retrospect, risk management strategies were called into question as they relate to software and system testing. And, the Securities and Exchange Commission (SEC) is in the process of drafting new rules following the Knight Capital debacle.
“I've seen many situations where automated backups failed to work as intended and when the company experienced data loss they found their backups useless,” said Sterling Jackson, a consultant at Affinity Informatics. “The ramifications ranged from losing a few days of work to shutting a business down permanently.”
Software to Blame for the Not-So-Friendly Skies
United Airlines was forced to cancel nine flights and delay 580 others when its backup systems failed in August 2012. Online, phone, and kiosk reservations were affected as well as connected systems that control departures and calculate flight load (passengers and baggage). Apparently, as with the Toyko Stock Exchange problem, a merger was cited as a factor. That is, the problem arose when the technology infrastructures of the merging companies were combined.
Manufacturing errors have also caused aeronautics angst. Teledyne-Continental Motors (TCM) had to recall about 3,000 crankshafts following a crash that caused minor injuries, which turned out to be no small task. First, the failure and the time period in which it occurred had to be identified. Once the manufacturing defect was identified, the company had to determine the scope of the problem and who was affected. Then, a number of esoteric Level III NDT inspectors had to be hired to do non-destructive testing (NDT) since the defect could not be confirmed visually; however, Level III NDT inspectors are few and far between. When the faulty parts had been identified a number of other problems arose, not the least of which affected the availability of new and factory-refurbished parts. In the midst of the crisis, a number of customer-service issues arose having to do with the recall and parts availability.
Drug Manufacturers Inadvertently Ship Drug Cocktails
Imagine buying an over-the-counter drug only to discover the pills inside were not as expected. Earlier this year, Novartis AG voluntarily recalled lots of Bufferin, Excedrin, Gas-X, and NoDoz because the pills were chipped, broken, or combined with other drugs as the result of a “packaging error.” While the company claimed consumers had not reported related health issues, the error nevertheless resulted in a one-time charge of $120 million. Operations were suspended at the Lincoln, Neb. facility for seven months for maintenance and upgrades.
The Federal Drug Administration (FDA) later reported that some of the pills may have been mixed with several opiates including Percoset. Other drug manufacturers have mislabeled products, mistakenly mixed unrelated drugs, and accidentally combined different doses of a drug in the same package. Apparently, Novartis had a long history of failing to report related customer complaints to management in a timely fashion.
“You need to be in a position to do something before it gets out of control,” said a manufacturing CIO who requested anonymity. “Sometimes that means doing some sort of analysis that no one has thought of to discover patterns that are causing problems.”
Marketers Demand Manual Overrides
In today’s online world, trades and media buys are executed at light speed using algorithms. The benefit of marketing automation is the ability to execute campaigns, such as pay-per-click advertising campaigns, based on a pre-defined set of business rules. The automation allows marketers to manage thousands of search keywords across multiple campaigns simultaneously. Automation tools also help speed campaign management times by an order of magnitude, such as weeks to days or days to hours.
The problem is: Automated buys left unchecked can cause serious budget issues in a hurry, whether it’s paying too much for keywords or allowing a campaign to run unchecked during time periods when the return on investment (ROI) does not justify the cost. Lou Susi, a senior user experience and innovation architect at Mobiquity said he chuckles when automation comes up in a business pitch.
“I think automation backfires quite regularly, actually,” Susi said. “The value proposition of the organization is typically some amazing formula [such as] ‘We automate _ blank_ process to drive down costs, build efficiencies and ultimately make for a better _ blank_ without all of that pesky human interference.”
Then, reality sinks in. Later, people discover they hadn’t thought the process through and so the solution isn’t fully automating the workflow. Three or more people may be required to make the process appear to be automated which doesn’t reduce the cost after all. The problem needs to be fixed but clients or customers outside the organization are relying on it so process re-engineering is difficult at best, Susi said.
How to Avoid the Homer Simpson Effect
In 20/20 hindsight, the forensic answer to automation is that testing, testing, and more testing should have been done earlier to avoid the crisis in the first place. However, while testing is important, it is not always the end of the story.
Manufacturers add process control. Businesses of all kinds use analytics to determine when software or manufacturing processes are operating “out of bounds.” Dashboards display the data in simple graphical formats such as charts and gauges indicating that action is necessary.
Sometimes, though, business leaders need to take a step back.
Sudhir Patil, a director at Qualitia Software, said at a more strategic level, there are five things organizations should beware of:
- Failing to have the organizational readiness to automate but doing it anyway because it’s a trend.
- Believing that procuring an automation tool is an end in itself. Know-how and supporting strategies are also necessary.
- Failing to understand what the expected benefit of automation is and how soon the benefit will be realized.
- Lacking the organizational agility to cope with the changes.
- Failing to understand what should be automated and should not be automated.
“At the end of day, the loss may be not only monetary but you may suffer a loss of confidence for all wrong reasons,” Patil said.
See also:
