Simplifying Hit Calculations for Xignite Web Services
from Xand : Joel York - 31st December 1969
The opinions expressed by this blogger and those providing comments are theirs alone, this does not reflect the opinion of Automated Trader or any employee thereof. Automated Trader is not responsible for the accuracy of any of the information supplied by this article.
At Xignite, most of our Web service subscription plans are quoted according to a uniform usage metric known as "hits." Estimating the number of hits you need for your application is straightforward, once you get the basic idea. However, for the first time customer it can be a little cryptic. So, we're providing this blog series to help you through the calculation process.
The first post in this blog series provided a comprehensive hit calculation tutorial and provided a hit calculation spreadsheet that covers all usage scenarios and introduced the following formula for calculating monthly hits:
Monthly Hits =
Hits per Data Block x Request Population x Request Frequency x 1 month
It's a great general reference, however, the vast majority of applications fall into a few standard usage scenarios where it's possible to dramatically simplify this formula. This second post in the series surveys these standard usage scenarios and guides you through the process of simplifying the hit calculation to something you actually can do on the back of an envelope.
It Doesn't Need to Be Perfect
First, here's a word to the wise. Xignite subscription plans span an enormous range of usage scenarios, from 600 hits up to 60,000,000 hits and beyond. For most Xignite Web services, each subscription plan is separated by a 10X factor in usage, i.e., 6K, 60K, 600K, 6M, etc. That's a huge step between each level (and a huge volume discount)! Therefore, your hit calculations don't need to be anything close to perfect. And, if it turns out your usage is different from the original estimate, you can simply change your subscription plan when it comes up for renewal. Hit estimates only require back-of-the-envelope estimates that are accurate within a factor of 10X and can be done in minutes, so you can decide which subscription plan fits your needs. The tips that follow will help you do just that.
Identify the Request Frequency Scenario
For most applications, the request frequency is the single most important factor in determining the subscription plan required. While the number of hits per data block may vary from 1 to 100 (e.g., single option price vs. a long option chain), and the request population may vary from 1 to 10,000 (e.g., single stock symbol vs. the total stocks on an exchange), the request frequency can easily vary from 1 to 10,000,000 (e.g., monthly downloads to 4 times per second cache refreshes). However, this huge range of request frequencies can be reduced to a handful of common trigger event scenarios.
• End User Page Views
• Transactions Processed
• Real-time Cache Refreshes
• End-of-Day Historical Data Updates
• Ad-hoc Downloads
Zeroing in on the event that triggers a Web service request allows you to simplify the general formula above into a natural format that relates directly to the usage pattern of your application. Calculating the request frequency is then a matter of estimating the average number of trigger events that occur per month as follows.
Monthly Hits =
Hits per Data Block x Request Population x Page Views per Month
(or Transactions per Month, Cache Refreshes per Month, etc.)
Real-time cache refreshes usually generate the highest request frequency. For example, a cache of real-time quotes that updates every second, 8 hours a day, 20 days per month (i.e., during market hours) will have a request frequency of 567,000 refreshes/month = 60 seconds/minute x 60 minutes/hour x 8 hours/day x 20 days/month. This is a good back-of-the envelope example, because even if you are only caching a single symbol, it is clear that a 600,000 hit subscription plan is the lowest possible solution. In the case of end-of-day updates to a historical database, the frequency is simply the number of working days per month, usually 20 or 30 depending on the application. And, ad-hoc downloads can be as little as one per month for reference data or fixed rates that do not change very often.
Simplify the Data Block
After selecting your trigger event scenario, the next step is to choose the simplest possible data block to make the hit calculation as easy as possible. In general, choosing a data block that equals one hit vastly simplifies your hit calculation, because it equates each hit with the natural data unit of your application, e.g., a symbol.
Single Hit Requests
Xignite pricing is very uniform across all Web services, so most hit calculations can be done with a simple, natural data block that equals one hit per request. For example, most quote, price and rate Web service operations cost one hit for each symbol across all types of asset classes (i.e., stock, currency pair, metal, interest rate, etc.) When the hit calculation uses a single symbol data block, the formula above reduces to the following.
Monthly Hits =
1 Hit per Symbol x # Symbols per Request x Request Frequency x 1 month
Monthly Hits = # Symbols per Request x # Requests per Month
For example, the hits required by a stock index widget would equal the number of indices in the widget (symbols per request) times the number of page views per month (request frequency).
Choosing a data block that equals one hit vastly simplifies your hit calculation, because it equates each hit with the natural data unit of your application, e.g., a symbol.
Even if your particular implementation relies on operations that take multiple symbol inputs, such as the GetQuotes operation of the XigniteQuotes Web service, you can still use the single symbol, single hit request data block as the basis for your hit estimate, because the pricing is uniformly one hit per symbol and will be the same whether you request multiple symbols at a time using the GetQuotes operation or one symbol at a time using the GetSingleQuote operation.
Varying and Random Hits Requests
Occasionally, your application may make use of operations that do not have a fixed, uniform hit count per input symbol. For example, you may use an operation like the GetRateFamily from the XigniteRates Web service where the number of hits per request varies by the number of interest rates in the particular rate family requested, e.g., Libors, Euro Dollars, etc. Or, you may use an operation like GetEquityOptionChain of the XigniteOptions Web service where the hits per request varies randomly for each input symbol depending on the number of options contracts in the option chain. In these cases, you can dramatically simplify the hit calculation by using the average hits per symbol from the average output data block (e.g., average # of interest rates per rate family, average # of option contracts per chain, etc.) When the hit calculation uses the hits for the average output data block per symbol, the formula above reduces to the following.
Monthly Hits = Average Hits per Symbol x # Symbols per Request x Request Frequency x 1 month
For example, the hits required by an option price research application would equal the average number of contracts per option chain (average hits per symbol) times the number of options reviewed each day (request frequency), times the number of working days in 1 month.
Take a Swag the Request Population
It's best to approach the request population calculation with the end in mind. If you've identified your natural trigger event and chosen the simplest possible data block, then the request population is always measured in [data blocks per trigger event], e.g., symbols per page view. The goal then is to come up with this number.
Request Population = # of Data Blocks Requested per Trigger Event
If you use the single hit data block described above (e.g., symbol, currency pair, fundamental, etc.), then the calculation reduces to the following.
Request Population = # of Symbols (currency pairs, fundamentals, etc.) per Trigger Event
However, the request population is very dependent on the application and there are very few rules-of-thumb to simplify the calculation further. Often the request population must be built from smaller pieces, e.g., the number of symbols in each widget times the number of widgets per page, etc. And, sometimes these pieces are random, e.g., the number of symbols in a user's portfolio widget. That's the bad news. The good news is that you only need to get the request population estimate within a factor of 10X to choose the right subscription plan! So, it's not a problem to gloss over the details.
Fixed Request Population
Many usage scenarios entail a fixed, unchanging request population for each trigger event. For example, a daily end-of-day update to a stock fundamentals database will have a fixed request population equal to the number of stocks in the database times the number of fundamentals tracked for each stock. Whoa, not so fast. How did you get that? You get it by beginning with the end in mind. The XigniteFundamentals Web service counts hits uniformly across operations as one hit for each fundamental data point returned. Therefore, the single hit data block is one fundamental. Applying the formula above gives the data population as the number of fundamentals per daily update. This in turn is equal to the number of stocks in the database times the number of fundamentals tracked for each stock.
Random Request Population
Undoubtedly, the most difficult hit calculations occur as a result of a random request population. How can you calculate the number of hits for an EOD portfolio accounting system when the number of stocks in a user's portfolio varies randomly by user from one to a hundred? It's easy! Just take an average and treat it like a fixed request population, because you only need to get it right within a 10X factor to choose the right Xignite subscription plan. If the average user has twenty stocks in her portfolio, then the data population is 20 symbols per daily update.