QA with application using chargeover



  • Our UAT uses chargeover for payment and other user management features. A couple weeks after we enabled automated testing about 50 test cases running about 2 runs every hour (can peak to 6-7 runs when more people are doing pre-commit validations), we got a call from chargeover saying we are making too many API calls to their service (10times/sec when peaking).

    Now, what can we do about this? We are already paying for commercial. Is there a way to set up a fake CO instance for development and testing? (probably not)

    PS: This is for integration testing, not API testing. The calls are made by our back-end services behind the scene.



  • I can provide a bit more detail here -- I work for ChargeOver.

    Firstly -- Please do feel free to reach out with us directly! We haven't spoken with you directly (we've spoken with Alvin about this only) but are more than happy to help in any way we can! Really! We love to help and code and want to make sure things are going well for you, and also don't impact our other customers negatively.

    Our concerns revolve around the really unusual profile/patterns of the API requests and the high volume of API requests. Currently we're seeing much higher API usage than just 10 requests/second -- as of this morning:

    • 110,000+ API requests per hour
    • average of 30+ API requests/second
    • peek well over 60+ API requests/second

    Some specific counts:

    2016-05-31 06am CST     119,206 requests/hour
    2016-05-31 05am CST     130,273 requests/hour
    2016-05-31 04am CST     106,377 requests/hour
    2016-05-31 03am CST      65,089 requests/hour
    2016-05-31 02am CST     112,494 requests/hour
    2016-05-31 01am CST     117,621 requests/hour
    etc. etc. etc. 
    

    What triggered some warnings internally here is that you only have roughly ~120 subscriptions (active customers) you are testing with, and ~5000 customers you are testing with, while doing ~100,000+ GET /customer/X requests/hour.

    Why do your tests need to query for customers ~100,000+ times per hour when you have only ~120 active customers?

    With regards to your specific questions:

    Now, what can we do about this?

    • Caching - Implement some level of caching in your application -- even caching a customer for 2 or 3 seconds is going to make a massive different in the number of API requests, and will offer the additional benefit of improving application performance on your end too!

    • Profile/examine your code - I suspect something is a little wonky within your app, given that you're querying for the same customers over and over and over again thousands of times every hour.

    • Talk to us - If you can reach out to us with more specifics, we'd be glad to look over your code, or discuss other options if you really need this volume of API requests. I don't want to really talk about pricing in public given it's private financial information about your company, but there may be options here if you reach out to us.

    Please let us know how else we can help -- we are happy to help in any way we can (but also need to make sure your unusual API usage doesn't impact our other customers).

    We are already paying for commercial.

    We may have options here if you reach out to us -- without getting into specifics, you are on one of our smallest/lowest pricing tiers.

    Is there a way to set up a fake CO instance for development and testing? (probably not)

    You are already making these API requests to a copy of your production environment (a staging/test environment).



Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2