Andy Webb: How would you describe Bloomberg's programming approach when you build infrastructure?
Shawn Edwards: Well, we definitely use an engineering approach here at Bloomberg. We build component-based software design in the infrastructure group; and we have a test driver for every component. You run it through memory profiles while you're running your test driver constantly.
To this day, especially in our core infrastructure groups, we look for people who have a real grounding in how computers actually work. You have to understand what's really going on underneath. So when we talk with somebody, we ask: "What's really happening when you make a call to a virtual function, or when you try to pass some information to another thread? You've got to learn about contention. You've got to learn about wasted cycles. You have to learn about all that.
As a result, we're always looking for people for the infrastructure team who understand about computer architecture, and what their code is actually doing. Once you know that, a lot of things can be built on top of that knowledge.
Andy Webb: When you're recruiting people, are you also thinking in terms of, for example, recruiting somebody who's got a firm background in really mission-critical stuff like flight-control systems? Where a mistake literally costs lives; where people have to work at that level of reliability?
Shawn Edwards: I have to break this down into groups.
To start, our core infrastructure teams are the people who are building the ground systems that our users rely on. These individuals have, as you say, experience in "mission-critical systems", which are built to deliver at high performance. We build our own databases here, market-data distribution systems, and more. So people with say, telecom experience, are great because they've had to think about reliability.
So, just as you said - people in mission-critical systems, but also delivering high performance. Your idea of flight-control systems is exactly a good place. You don't find a lot of those people.
On the other end of the spectrum, a big part of what we do on our
infrastructure team is building the application framework; we're
building applications all day. We've hired people from the gaming
industry for these roles. Their high-performance UIs are using
scripting language on top of native C and C++ code. So it's
exactly the right model, and they get it. They know where the
boundaries are, and what the
Bloomberg's a big place. We have different groups with different needs. For many application groups we like to hire people with say, finance backgrounds and who are technically savvy enough for us to teach them the higher-level programming languages. They go out to see customers and iterate on prototypes, building our UI workflow, which is such a critical part of what we do.
People are typically good at a couple of things. It's hard to find somebody who can build, say, super-high-performance market-data systems or trading algorithms and who is also good at building incredible UIs. There's usually a knack for one of those things, but it's rare to find a candidate with both the technical knowledge and the industry experience. Bloomberg aims to hire people with a diverse professional background and solid programming skills.
Andy Webb: I believe your own background programming-wise has been predominately C++?
Shawn Edwards: Actually, I started off in the hardware design world as a logic designer, building mainframe systems at IBM. I switched over to writing software and building simulators and routing algorithms in CAD systems for Mentor Graphics. That's where I got hooked on software development as a career focus. When I was at Bear Sterns just prior to coming to Bloomberg, my focus was pretty much all C++. So, you could say, I spent most of my programming life in C++.
So in this programming environment we give our developers seamless access to all the high-performance services. Those services are all typically written in C++. There is a large catalogue of services, all of it riding over our own middleware. Our goal for Rapid was all about shortening a product's time to market and about providing an environment where we can iterate on concepts. We're huge on iterated design process at Bloomberg!
Andy Webb: That's very interesting ...
Shawn Edwards: You can't think of the Bloomberg terminal like a Web browser. Our users have four, six, eight monitors running at a time. They have data splashed across every screen; charts, tables, news, real-time analytics running on multiple screens and interactive functions. All of that is being powered by this rich desktop experience, which is being orchestrated by this Rapid layer, which is calling out to services. So, from the Rapid programmer's point of view, the services that a user is running can be anything. Programmers call these interfaces and leverage a wide range of powerful interfaces.
Bloomberg's User Experience team, led by Shawn Edwards, built a state-of-the-art Usability Laboratory (above) at the company's New York headquarters. The team uses the lab to test new features and design elements by asking clients to review design enhancements and provide feedback. "The design process here at Bloomberg is iterative and ongoing," says Edwards, "Testing design elements early-on in the product development process gives us confidence because it gives us facts over our own opinions. There is no substitute for a real customers' feedback," Edwards explains.
Andy Webb: How far into the future are you projecting your planning? Do you segment your road map and think of long, medium, short term?
Shawn Edwards: We see various stages to road-map development. For example, building Rapid has been a multi-year effort. We started four years ago with a vision. We know which direction we want to take it, but we need to have more than just a vision. A lot of the details are worked out as we go along. Bloomberg always aims to be nimble enough to evolve with a road-map and never too bought into our own ideas that we can't adjust the development plan when necessary.
We've been talking about a few things recently, namely the Bloomberg open symbology and the Open API - both were not a three-month project, but longer-term initiatives. The Open API is not typical of what we normally do. We normally build our systems and our infrastructure based on our interactions with our customers. Now we're saying: "Okay, hey, we want to work with the open-source community, and other vendors."
I have an idea of where the Open API should go. But once you work with other people, you have to listen and absorb feedback. You have to say: "Okay, let's see where it goes and what things we learn." And it will be community feedback as much as what continues to work with our development philosophy that charts the course.
At the same time, other projects are more planned out and concrete out of necessity. We certainly have a plan for what we are doing in the next six months and this year, for example. But those concrete plans co-exist with more fluid long-term visions as well.
Andy Webb: When you've got time off from the day job, do you spend time picking up on technology that's obviously very bleeding-edge now but just not ready for you? Filing it away in the back of your head because, say, there's this thing that you might be doing in about a year's time, and it could fit in there? Do you have any sort of formal process, or is it just an informal process?
Shawn Edwards: I've never had a formal process, but I do a lot of reading and a lot of talking with people on my team who are a great source of information and learning. A big part of my job is making sure we build the right teams; because often the best ideas are really coming from the team. I'll give you an example. It wasn't my idea to use GPU computing, someone on the mortgage team raised his hand. He said: "Hey, let me go and prove this." And we said, "That's a cool idea. Prove it." He contacted NVIDIA and got some of their early cards. And he proved it to us; he actually went ahead and built our entire GPU computing farm, which we're still using today.
Big picture: part of my job is finding and hiring the right people for the right positions and also listening to the team's collective knowledge. For the most part, the hot-bed of innovation that is Bloomberg is the work of more than one person - in fact, it's more than 3,000 people!
Andy Webb: That was what - back in 2009, wasn't it?
Shawn Edwards: Yes. An analogous thing today would be some of the emerging big-data approaches. There's a lot of hype. In every wave of technology, there's more hype than there is real stuff. And we're looking to see what's coming out of those concepts that's useful.
We're doing interesting work ourselves such as tying together structured and unstructured data in a way that can deliver answers really efficiently and more succinctly. More and more people are looking at more complex interactions in the markets. We think that anything we can do to help them, is important. We're constantly talking to people and listening and then it's up to us to determine what comes out of those discussions that are useful in the long-run.
Andy Webb: I have a question about the GPUs. You made the announcement back in 2009. But obviously your guy must have started working on it sometime before. At that time, I wouldn't have described Tesla and NVIDIA as having the hardware necessarily bulletproof, nailed-down, finally ready. So you were taking - I wouldn't say a gamble exactly, but you certainly weren't waiting for everybody to take all the risk first.
Shawn Edwards: Absolutely. NVIDIA gave us some single precision floating point chips. We needed the double precision floating point chips. They gave us a lot - they were fantastic to work with. We have worked with them on their product, their development environment, debugging, etc. for several years. They gave us a lot of their engineering time and we had a really healthy back and forth.
Of course, we try things, but not all ideas succeed. If you succeeded with everything you try, you're probably not taking enough risks. But at the same time, we have to be careful, too. I think there are more fads in the tech industry than even the fashion industry. There's a lot of hype. There's a lot of noise. And it's our job to kind of cut through all that noise and see what's real.
A lot of what Bloomberg has built that's really interesting, we built ourselves. We've been forced to build it ourselves because of the scale and the performance needs that Bloomberg has; whether it's from the ground up, or starting with some open-source software or sometimes part of it is a commercial package. We're very much a 'build - vs. - buy' kind of shop out of necessity because we have a unique architecture and because we do things differently.
But we are always looking and trying things. It's just part of what we have to do. It's always based on a purpose however. It's always based on solving problems for our customers. When we have a problem, it's typically 'this is what the customers are trying to do, how do we help them do that.' Or, we have a vision of what would really help them and we're building the solution with that vision in mind.
Andy Webb: When you latch onto something and you find that it works, do you look to redeploy it elsewhere?
Shawn Edwards: The natural extension would be to other analytic groups in Bloomberg seeing a need, for somebody else in the back end doing some other type of calculation, say risk calculations.
The kind of extension you're talking about is more of a departure. How do you open up the user's desktop, let's say - and this is what I infer from what you're saying - so that Bloomberg software could utilise the GPU that's on their desktop? That's kind of interesting because we're always debating how much to do on the desktop versus on the back-end.
Like I said, we have a special client. It is rich in the sense that we try to maximise whatever power we have on the desktop. The only problem with taking advantage of the GPUs on the desktop is that it's not uniform. We have 300,000-plus subscribers to the Bloomberg Professional service. We're very cognizant that Bloomberg should just always work everywhere. So that's why our back-end model's so powerful for us. So we may instead take the algorithm and put it in the back-end and do some calculations and send the results back.
At the same time, we're always debating how much to do on the client side. And we're doing some really unique, cool things on it even in Rapid. While Rapid sounds like it's just a UI framework there's this whole native component for high-performance communication in a whole data cache layer, which is more than a cache. It's allows for pivoting and complex aggregations all in native code; a UI layer on the client side just sits on top of that. So, we're taking advantage of the CPU as much as we can.
Andy Webb: A little quip I came across on your recruitment site. It says: "agile with a small a," which is kind of nice. Do you reckon that's always going to be your way?
Shawn Edwards: Time to market is huge for Bloomberg. If there is a good idea, we want to get it out to our customers fast. And then we can improve upon it and add the bells and whistles and add the right features and functions in a way that complements our customer's workflow. We want to get it out there. It's two-fold. One is: the users are getting some value out of it immediately. They need that insight into the market. Then we get feedback from our customers and make it better.
Instead of first building the perfect system, which will never be perfect without the user's feedback; we don't wait for that. We get something out and then we iterate on it. We don't have rigid processes, so agility is fundamental to what we do. That's what we mean by 'agile with a small a.'
When you're building infrastructure and you are building, let's say, ticker plants and market data appliances, the level of agility is different than when you're building a screen for somebody. We put out an analytic screen, called DEBTthat displayed the U.S. debt ownership that both the banks and sovereign nations were holding during the financial crisis - we put that information together on a single screen in a weekend - and it was correct. The data was valid. We could ensure that that the design and the data was high quality - and people loved it when we rolled it out. It was the one place that you can see all that information and access it in an intuitive way. You really can't do that with some other types of systems.
Andy Webb: With that approach, do you find that occasionally there are generic testing harnesses that don't necessarily work as you would like? Do you end up having to build a lot of the proving tools and testing tools for your way of doing things?
Shawn Edwards: Let me back up a little bit to say, the infrastructure teams build these environments: the Rapid environment, the databases, the market data, and this entire environment that the application teams build within. That suits their needs 98 percent of the time - or some high-nineties percentage, at least. As long as they're in that environment, as long as they're within the platform so to speak, the Bloomberg product, they have incredible agility.
It's only when somebody's coming up with something different that our infrastructure doesn't support, when they're pushing the boundaries on something. That's when we have to think: how do we extend the architecture? How do we extend our UI toolkit, or our widget set, if you will? For the most part, we are trying to make the vast majority of Bloomberg developers super-productive. When they are trying to push the boundaries, we always take step back and ask if we need to build something for them.
As far as test tools and harnesses, because we're building this infrastructure and this platform for the large majority of programs - again, there's over 3,000 technical people at Bloomberg; so we have to have this uniformity. That's how you get economies of scale.
Andy Webb: Okay. But on the infrastructure team, what have they got as regards testing and tools and things like that?
Shawn Edwards: They're building it. Everything from building component-level test drivers on C++ code to building replay services for ticker plants to testing out various low-level C routines.
I don't want to say we write every line of code, because we don't. We are constantly looking at using open source code and we have some really great partnerships with vendors.
Andy Webb: I'm just intrigued by the fact that you don't accept the status quo, the generic technology that's out there. I'm not quite sure if it's a Bloomberg design, the IDE design - is that your version of Visual Studio for Bloomberg or what?
In addition, Bloomberg looks to hire people with diverse backgrounds, and once we train them we see their amazing UI designs. They can build these incredible concepts, and do so incredibly quickly.
For example, I was once in a meeting with the head trader from a buy-side firm. There was this new workflow concept that the trader explained to us. It was a communication between portfolio manager and trader and groups of traders, a workflow among people - and it was hard to explain, so he drew it on a white board.
When we broke for lunch, I asked somebody to take a picture of the white board, and about an hour-and-a-half later, after lunch, we presented the trader with what he had drawn - as a function Bloomberg screen prototype. He was just totally blown away.
It is that kind of iterative cycle that we use internally. That's what we do internally with those concepts: paper prototypes, white boarding, and then Rapid prototypes, using those tools, those IDEs.
Andy Webb: Fantastic. One question. Your development team - how many guys did you say were in development technology at Bloomberg?
Shawn Edwards: The vast majority of people in Bloomberg R&D are programmers. The infrastructure team represents a subset of the programmers, approximately 10 percent of the 3,000 plus R&D staff.
Andy Webb: You mentioned you have loads of conversations with clients, often at a really technical level. Would you say that your client conversations have changed significantly?
Shawn Edwards: Yes, for some aspects of our business the conversation has changed and it has influenced Bloomberg's focus on providing enterprise solutions to our customers.
In the last several years, the markets have expanded from a regional perspective and an asset class perspective, but it's contracted in many other respects. Therefore, our customers' needs have changed as well. For the most part what we see is that our customers now have to do more with less. So, we started talking to our customers about what more value we can bring to them.
When we were talking to customers, we noticed they're committing a tremendous amount of effort and IT spend to building out their own infrastructure. In today's world, building your own feed-handlers and ticker-plants and running your own huge market data systems is no longer delivering a competitive edge, but it represents a real spend. What came out of those conversations was the evolution of Bloomberg's EP&S, or Enterprise Products & Solutions.
Bloomberg is the original managed service provider for the financial markets, because we have always delivered our products as a managed service. Today, Bloomberg desktops and data are integrated into our customers' business processes. We can play and even bigger and more valuable role supporting the customers' infrastructure and at the same time help them grow, even when IT budgets are shrinking.
For instance, we've talked to customers about building algos and the cost and the barriers to entry to get into different markets, and operational support models and the headaches they have integrating with different vendor solutions. You know, it's not always about squeezing out nanoseconds. It's about them having to hire people just to manage infrastructure. It's a valuable conversation when you get to step back and say: "What are you really trying to do?"
Andy Webb: Yes. "Do you want to trade? Or do you want to manage the plumbing?"
Shawn Edwards: Exactly. When I speak with our customers about this subject, I hear almost the same story over and over again.
Vendors in the low-latency market, the direct-feed market; they're regionalised. So if they have a site in one city in the US and another in, let's say, London, and they want to move over to Singapore and Sydney - it's an explosion of different APIs, different vendors, different SLAs. That's what's impeding these guys from testing out new markets. If there was a lower barrier to entry - and it's not just the dollars spent; it's actually the time it takes in debug the errors that you get. I heard a lot of feedback from people who are tired of building plumbing.
Andy Webb: Shawn, thank you so much for your time. This has been a fascinating conversation.