What kind of usability testing can I perform when the final users of the product are developers?



  • I have built a framework aimed at developers and I'd like to carry out a test with users. However, since my final users are developers, I don't know how to approach this testing. The usual list of tasks to perform doesn't seem appropriate in this context.

    Some context about the tool:

    There are several APIs out there having a similar functionality but being different enough so it's not trivial to use them together. I've developed a framework that allows the user to forget about this integration, focusing on the results returned by the APIs combined. This way, developers don't need to spend too much time integrating the different APIs into their system, they just need to implement a common interface (if it's not implemented yet), launch my framework in a server of their choice and send their requests to it.

    Considering this, is there any kind of test which fits best for this situation?

    In the end, I want to test if using my system is a better/easier alternative to integrate the APIs by hand. I don't know if I should make a list of tasks to accomplish using the tool and then ask some questions, or if I should make someone code something using the API and without using it and compare the resulting codes (checking static code metrics, for instance), etc.

    Any help is appreciated.



  • Interesting task. You could run two usability tests with developers:

    • one with developers using APIs
    • one with developers using your framework

    and collect statistics (like satisfaction on Likert scale) and feedback on challenges (from think-aloud protocols) from both of them and compare. You care about both quantitative data (how good your framework was) as well as qualitative results (at what parts it was better and where it still needs some improvement and why). I once collected a lot of useful feedback about Web service search engine by just sitting together with 16 developers (in a separate 16 sessions) as they were using the assigned search engine to find a Web service and listening what they want to do and what they struggle with (this is form of thinking-aloud protocol).

    There are many parts that contribute to how usable an API or framework is: design/architecture, examples, documentation, IDE support. Microsoft has performed usability tests for one of their libraries.

    I have also ask similar question about this topic here.


Log in to reply
 

Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2