Testing of distributed and asynchronous processing



  • We have camel based system domain-agnostic data channel framework which facilitate the raw data flow from an external data source , validate the raw data, enrich and standardize the data to ensure that data can be consumed by our system, determine the changes to the data and persist the data. All of these asynchronously and exposed as rest webservices. We need to develop a test framework for this system. Please suggest tools and design for this.



  • I have worked on systems which involved (Async Processing). The implementation of Asyc processing will be technically achieved through below examples

    • Queue based processing (Service Broker / MQ Series)
    • Custom Tracking tables (JMS etc..)

    In all this cases, You can either validate by having

    • Retry for fixed number of times with configurable timeouts to validate messages are processed. I have seen a similar case in file processing, file will be picked up but it will not be loaded completely (processing). In this scenario I wait and retry for a configurable number of times to validate data loaded
    • Option#2 - If you have any tracking table you can run your validation once tracking table notifies messaged is processed
    • Option#3 - Wait until all messages are cleared from the queues, directly query where the message queues where messages are stored intermittently to identify message status


Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2