It was a Friday afternoon when the contractor - we'll call him Joe - seemed to be behaving particularly strangely. Joe had been with the fund for a while but had not been performing well. The hedge fund manager couldn't put his finger on it. Perhaps it was the odd mannerisms or the abrupt way Joe had announced plans to do some work from home. But the manager grew suspicious, so he and a trusted inner circle of staff started checking the logs.
"Sure enough we saw that he had essentially accessed one of the traders' machines," the manager said. "He tried a bunch of different passwords and user names but eventually got in. It was clear that he had made copies of stuff that he never should have made copies of."
Careful not to jump to conclusions, the manager and his small team spent all of Friday evening gathering evidence. That stretched into Saturday as they combed one machine after another. Finally on Monday they met with the police and brought in legal counsel.
The fund, which shared its experience with Automated Trader for this article, has asked to remain anonymous, for obvious reasons. The events leading up to and following the detection of the attempted theft illustrate just how fraught an issue network security can be. Firms that spend vast sums designing trading models and amassing data face not only a litany of threats but also a wide variety of choices for what do if there is a breach.
"We used to think of security in terms of reducing the chances of unauthorised users gaining access to critical systems or data," said Jim Doherty, chief marketing officer at Certes Networks.
"I think we have to start assuming that eventually these efforts will fail or, as we saw in the Goldman affair, that a trusted insider will turn on us. Given that 'inevitability', companies should be thinking about how they can limit the damage any one breach could cause. If you assume you can't prevent bad things from happening with 100% effectiveness, then the next best thing is to do your best to minimise the potential impact of those events."
Doherty was referring to a public case involving Goldman Sachs which is on-going and being watched for its legal implications in the United States. Although the scale of the Goldman affair is much larger, there are parallels between it and what happened with Joe, whom we'll be hearing more about later.
Sergey Aleynikov was a computer programmer for Goldman for two years who had done work for the group's high frequency trading activities. He was accused of stealing "thousands of lines" of code during his time there, right up to his last day of work. The files were allegedly encrypted, sent to his home computer and then stored on USB flash drives and a laptop.
He was charged under the National Stolen Property Act and the Economic Espionage Act, and ultimately sentenced to eight years in prison. But the ruling was overturned on appeal as Aleynikov's lawyer successfully argued that these laws didn't apply due to the intangible nature of what had been taken. He is still fighting state charges in New York for unlawful duplication of computer related material and unlawful use of secret scientific material.
Doherty said the main take-away from Goldman is that companies need to consider worst-case scenarios to understand not only where they are most vulnerable, but also where and how the greatest damage could be done in the event of breach, whether by external or internal actors.
Mark Sangster, director of marketing for eSentire, said he expects employee monitoring will have tightened in the aftermath of the Goldman affair.
"The fact that the appeals court ruled that computer code does not qualify as stolen goods under federal law has enormous implications on how firms protect their 'secret sauce' such as high-frequency trading algorithms," he said. "This may mean looking for legal protection or remediation through other laws by patenting algorithms and using patent infringement laws."
Network security experts say there are a range of steps companies can take to minimise risks, though they still encounter firms that are not implementing even basic security measures. Efforts to combat security risks can fall into three buckets: what firms do to prevent any breach, how illicit activity is detected, and what a firm does to contain the threat once it has been detected. Threats can come from outside or from within, and it's the latter that often pose the biggest risks.
Bucket Number One: preventative medicine
Sangster stressed the importance of background checks on any employees involved in programming a firm's "secret sauce". He also advised creating a security hierarchy for code creation, and granting access to algorithms to as few necessary programmers as required.
Code should remain within a controlled 'sandbox' which prevents its expropriation, he said. "This also means controlling access to the code tree through development tools, and closely monitoring the check-in and check-out process," Sangster said. For example, only people on a build team which assembles code modules in a finished product should be able to assemble final code and place it in specific - and monitored - locations.
"It's not a pleasant process to assume the worst in your employees and work to limit the damage any one employee could cause," said Doherty of Certes. "But it may save your company a lot of pain later."
In the case of Joe the code thief, the hedge fund manager said recruitment processes are tighter now than when the contractor first came to work for the firm. "He would not slip through the cracks now," the manager said.
A nice side-effect from devoting time to preventative medicine is that employees know that illicit activity would not yield much benefit. "Whether the motivation is monetary or personal, there is still a risk-benefits analysis that happens before someone acts. If they know up front that value (or the damage) is limited, it may dissuade them from acting out in the first place," Doherty said.
Having restricted access to information also pays dividends if the threat comes from outside the company, such as via hacking attempts or malware.
Steve Schoener, vice president of client technology at Eze Castle Integration, said implementing an access control policy to restrict access to unnecessary users is one of the key steps a firm can take to ensure network integrity when fighting off external attackers.
Bucket Number Two: detecting the breach
If the first category is all about being on the lookout for dodgy characters and maintaining sound information barriers within an organisation, the second bucket is more about technology.
Sangster said companies should have constant monitoring and quarantine service and should log all access to codes. Any movement of code to portable devices such as laptops or USB drives should be monitored and potentially blocked. The same went for cloud-based storage and email services.
"There are various means of detection that can be deployed - each matched to different types of 'markers' for illicit traffic or activity," Doherty said. "Most of the detection methods simply tell you what to pay closer attention to, and you want them to err on the side of having some (or many) false positives. It's better to have these systems tell you to look at 100 possible incidents with one actually being something serious than to shoot for perfect detection and have something slip by."
The hedge fund manager who caught our friend Joe, however, noted that even with good logging software, there is no fool-proof way to identify nascent risks.
"First of all, you can't log everything because the amount that you log will just be enormous," he said. "We do log a fair amount of stuff, but not everything is flagged up - because there'd be things going on all the time. It's just not realistic."
At the same time, if a firm logs selectively, there is the chance it is being too selective. So the software needs to be sophisticated and look for historical patterns and correlated events.
Doherty said there were always going to be trade-offs when deploying security tools. "What you want to look for in a security solution is how it performs with regards to manageably, transparency and flexibility."
Schoener of Eze Castle highlighted another tool: education. "A simple way of increasing network safety is by educating employees about fundamental security protocols."
He said the majority of attacks were undirected and came from opportunists scanning for any security weaknesses. "The real danger comes when a specific firm is targeted, in which case the attacker's efforts become more sophisticated, and they're much more likely to persist until they succeed."
Meanwhile, regulatory demands for increased reporting have pushed many funds to look at cloud computing in order to reduce IT costs, and this presents another set of issues.
"Companies going to the public cloud need to treat the cloud with the same level of respect and security as their on-premise systems and networks", said John Howie, chief operating officer of Cloud Security Alliance.
"They need to start with due diligence during the procurement process, ensuring they select a cloud provider who will take the necessary steps to protect their applications and data. Once into the cloud they need to maintain a comprehensive education and awareness programme to end users about the nature of attacks and threats, and what constitutes suspicious activity and who to contact if they believe something is amiss."
Howie said that in most successful attacks against a firm in a public cloud it was the firm itself that had failed to maintain security by divulging passwords as a result of phishing or other attacks. He argued that cloud computing can even offer security advantages for some companies.
"Cloud providers invest literally millions of dollars or more annually protecting their cloud infrastructure," Howie said. "It is unlikely that any company can match the resources that a cloud provider has."
Cloud providers typically employ an ISO-certified ISMS (Information Security Management System), coupled with a risk management program. Controls are typically documented and most cloud providers will share the reports with their customers under NDA. There is also a move in the industry to document how clouds are built and operated in an effort towards increased transparency and to reduce the procurement lifecycle, Howie said.
Along with cloud computing, another big trend is the use of internal Wi-Fi networks and a 'bring your own device' policy. But if employees are accessing files remotely, the problems from a stolen laptop or smartphone can become serious. The solution for many companies is to install software on all employee PDAs so that if one is reported as missing or stolen, it can be remotely wiped immediately.
Bucket Number Three: catching them red-handed
If, like the anonymous hedge fund manager, you succeed in spotting the rogue employee before he or she has walked out the door, there is then the question of how to handle it.
The first impulse for many funds, which prize secrecy, is to do everything on the quiet.
"Companies have this tendency, you know, to keep a low profile," the fund manager said. "File for charges immediately. They will go and arrest him. They will search his property. And the quicker you do that, the better."
The manager said he doubted that thefts like this were based on the idea of selling code and data to another party. Rather, the thief thinks he or she might be able to use it directly. "He sort of knew what he was looking for," the manager said. But in this case, the information taken would not have been useful. "There's a lot of stuff you would need to replicate in order to be able to use that."
The right balance
The main issue for firms, particularly smaller ones that don't have large IT departments, is in striking the right balance between being too lax and overkill. For instance, just because a fund may take a multi-layered approach to security, it needs to be practical and ensure that its operations don't get bogged down. So targeting the most vulnerable areas is key.
In the end, it comes down to a firm's philosophy. The man who caught Joe argues that for his firm it's better to have slightly lower barriers but better monitoring than to have extremely high barriers.
"Here's the thing, if you have very strong security and then somebody manages to bypass that, you're almost not going to catch that," he said. "If you've got very high-strength security measures, there are people who can bypass that and you definitely won't have the equipment to even figure out that it happened."