hourly limits / better filtering

Discussion in 'API' started by neoncat, Dec 20, 2013.

  1. neoncat

    neoncat Feline Outline

    (NB: thanks so much for putting out an API! I know it's a just side project, and it's hard to design a good API, so the following is just some helpful commentary after trying to do something complicated with the data...)

    Quick experimentation with the test api has shown that there needs to be either a higher request limit or better server-side filtering.

    Basically, the difficulty is that one has to submit a bunch of requests to the characters (C) or items (I) table in order to have enough data for a useful JOIN-SELECT with the battle (B) table. C and I both include data from single-player battles, which means that the relevant subset of data is relatively sparse compared to the kind of requests we can make. Thus, a range request (max of count=25) is only slightly more effective than requesting data for a single battle. If one constrains the subset of B further (e.g., battles involving a certain player), I and C become so sparse that one must request data for each battle individually.

    Here are a couple of possibilities (that y'all have probably considered... ^_^):
    1) archive data into static files segmented by day - this would let folks obtain large chunks of relatively recent data without overburdening the live database. The lower request rate currently in place should suffice for those few applications which need up-to-the-minute data, like tournament management.

    2) accept a json array parameter of battle ids - for tables C and I, this would permit a useful "lumped" request which the current range query doesn't offer (due to the sparseness of data in C and I). It would still be limited to 25 per request. This could load the database quite heavily, though, since it wouldn't be a sequential access.

    3) server-side JOIN-SELECT - I suppose this isn't really feasible due to the complexity and server load, but what people (or me at least...) really want is to filter C and I according to various characteristics of B (and eventually the player table, too). I'm happy to perform that operation on the client, but others may not be so comfortable with data manipulation.

    My personal preference would be option #1, because I just want ALL THE DATA, which I will then peruse at leisure over a cup of tea. Also, it's less work for y'all.
     
  2. Flaxative

    Flaxative Party Leader

    #1 would be amazing.
     
  3. neoncat

    neoncat Feline Outline

    Oh, I should also mention that if y'all go with something like option #1, I'm more than happy to make available to the community whatever tools I write for messing around with the data in a browser. That way, the community would end up with a client library that looks more like option #3 (from the end-consumer standpoint), but with y'all doing less work. ^_^
     
    ParodyKnaveBob and Flaxative like this.

Share This Page