[ad_1]
At 7:04 am on a fall Thursday in Tokyo, managers of the world’s third-largest stock market realized they had a problem.
A critical data device for the Tokyo Stock Exchange trading system had failed and the automatic backup had not been activated. It took less than an hour before the system, called Arrowhead, began to process orders worth US $ 6 trillion (RM24.95 trillion) stock market. Exchange officials saw no solution.
The full-day shutdown that followed was the longest since the exchange switched to a fully electronic trading system in 1999. It drew criticism from market participants and authorities and highlighted a less-discussed vulnerability in global financial plumbing, not the software or security risks, but the danger when one of the hundreds of pieces of hardware that make up a trading system decides to abandon the ghost.
“Exchanges are a crucial part of the market infrastructure and it is unacceptable that business opportunities are being denied,” Finance Minister Taro Aso told reporters in Tokyo. “These are machines, so it’s always possible for them to break. They need to build the infrastructure with that possibility of breakdown in mind.”
The TSE’s Arrowhead system was launched to much fanfare in 2010, and was announced as a modern solution after a series of outages in an older system put the exchange to shame in the 2000s. The “arrow” symbolizes processing speed of orders, while the “head” suggests robustness and reliability, according to the exchange. The system of approximately 350 servers that process buy and sell orders had had some hiccups, but no major disruptions in its first decade.
All of that changed on October 1, when a piece of hardware called shared disk device number one, one of the two square-shaped data storage boxes, detected a memory error. These devices store management data used on servers and distribute information such as commands and ID and password combinations to terminals that monitor transactions.
When the error occurred, the system should have performed what is called a failover: an automatic switch to device number 2. But for reasons that the exchange executives could not explain, that process also failed. That had a ripple effect on servers called information distribution gateways that are meant to send market information to merchants.
Disappearing data
At 8 a.m. M., Traders preparing at their desks for the market opening an hour later should have seen indicative prices on their terminals as orders were being processed. But many saw nothing, while others reported seeing data appear and disappear. They had no idea if the information was accurate.
A minute later, the exchange made its first communication, informing the systems administrators of the securities firms that there had been a problem. At some brokerages, that didn’t immediately leak to bewildered trading desks.
At around 8:05 a.m., Twitter, which is often used by merchants to communicate outside of the more official communication channels monitored for compliance, began buzzing with rumors of a problem. Traders described a growing sense of confusion as few responses came from the stock market.
“We didn’t know if it was our system or the exchange,” said Masaya Akiba, a broker in Marusan Securities Co.’s stock trading department. “We only confirmed this when the exchange released a statement.”
At 8:36 am, the exchange finally informed securities firms that trading would stop. Three minutes later, he issued a press release on his public website, albeit only in Japanese. A confusingly translated statement in English would not follow for more than 90 minutes.
It was the first time in nearly fifteen years that the exchange suffered a complete business disruption. The Tokyo Stock Exchange has a policy of not closing even during natural disasters, so for many on the capital’s trading floors, this experience was their first.
Historic decision
Some market participants were infuriated by the shutdown. Others, with nothing to do, occupied their time reading research notes or trading commodities.
“I didn’t think much of it at first,” said Kiyoshi Ishigane, fund director at Mitsubishi UFJ Kokusai Asset Management Co in Tokyo. “The previous outages were quickly resolved, so I assumed the orders would just be delayed.”
In 2012, after the move to Arrowhead, the exchange quickly resolved limited issues. Many expected the bag to do the same this time, too.
But as the hours passed, Hajime Sakai, the chief fund manager at Mito Securities Co, grew increasingly uncomfortable.
“I really couldn’t pay attention to much else,” he said. “I wasn’t like, ‘Open the market!’ It was more like, ‘Whatever, make your decision, quick.’
The call was daunting. After the failed switch to backup, the exchange had manually forced a switch to shared disk device number 2. At this point, administrators had a choice: they could try restarting the trade, but this would have involved a full system reboot , turning off the power and restarting.
The data of the orders already received from the securities companies would have been lost, without having been canceled. That would have led to anarchy, securities firms told the exchange. After talking to market participants, the exchange made a decision: trading would be suspended for the entire day.
Many in the market say they were relieved. A call to resume operations would have been chaotic, said a worker at a Tokyo-based brokerage, with no way of knowing which existing client orders remained active, while also trying to process new requests and offers.
Technical discussion
At 4:30 pm local time, four TSE executives, including Chief Executive Officer Koichiro Miyahara and Chief Information Officer Ryusuke Yokoyama, confronted reporters at the exchange to explain the outage. In a briefing that lasted approximately 100 minutes, they bowed in apology in front of the crowded room before getting into a detailed technical discussion about the breakdown.
If the exchange was criticized for its communications earlier in the day, it won praise for how it handled the press conference. Executives answered questions from the media with relative ease, discussing areas like systems architecture in highly technical terms. They also took full responsibility for the incident, rather than trying to shift blame onto the system vendor Fujitsu Ltd. It was little like the bug-filled reports of other Japanese companies in the past. On Twitter, the Japanese public expressed approval.
“Management explained very clearly during last night’s briefing,” said Megumi Takarada, a senior analyst at Toyo Securities Co. in Tokyo. “The briefing provided some assurance that management clearly understands the problem.”
Later in the evening, it was announced that the stock would be listed again on October 2. While that went smoothly, many questions remain unanswered. The Financial Services Agency ordered the exchange to issue a report on the outage, according to local media, which may provide more information on some of the issues.
But one of the most important is whether the same kind of hardware-driven failure could occur in other equity markets. For a strategist, it almost certainly could, but that’s not something to worry too much about.
“There is nothing uniquely Japanese about this,” said Nicholas Smith of CLSA Ltd in Tokyo. “I think we have to put that in the box of ‘things happen’. These things happen. They shouldn’t, but they do.” – Bloomberg
[ad_2]
