Thursday, April 28, 2011

User Evaluation - Session 1

Now it's time to test our system in a proper manner. We have practices to conduct our evaluation in previous pilots. And we have developed our application according to the prior feedback as well.

We have managed to gather 10 participants from University of Sri Jayawardenapura (USJP) with the help of Mr. Chanaka and Mr. Kasun Nayanajith. We have divided the evaluation in to two days (27th and 29th April 2011), thus five participants per day. We were lucky to acquire a sound proof room which is used for sound recordings.

There were 2 female and one 3 male participants. Among the female participants 1 was totally blind and the other partially blind. Among the male ones 1 was totally blind and 2 were partially blind.




We managed to set-up the evaluation environment by 9.30 a.m. on 27th of April and managed to start the evaluation at 9.40 a.m. As planned I was assigned to note down personal information of the users. After that Kanishka demonstrated the system and then the user is prompted to perform the given task. I have recorded the time, errors and memory aids provided while Malshika handled the recording and making notes. Afterwards the user is asked to describe the layout of the home page as he/she remember and then the home page page was replayed to assist the user to describe. Finally there was a Q&A session which had 14 ranked (1-5) questions, which was handled by my self.

According to our observations and discussions we got various types of feedback on the system; positive, negative and medium.

Positive feedback were generally on the;
  • Identifying different objects on the screen
  • having uniform positions for particular functions
  • Error recovery
  • Table structures
  • Data comparisons

Among these; ididentification of the table structures gained high amount of positive responses. They said it is very difficult to distinguish among different rows and columns on a table when they are using a screen reader like JAWS even though it says that this is a "Table".

We have observed that users had very negative responses on the speed and clarity of the speech and said that it would be better to increase the speed and clarity.

We got medium responses for the points like;
  • Navigational layout
  • Interactivity level
majority of the users identified the locations of the objects on the home page very accurately though they were not much successful in recalling those location s from their memory. One of the participants even turned his head towards the sound directions as we noticed.

Typing speed was another important factor in the evaluation. Some of the users showed significantly low completion times since they are well trained in typing. One of the users have used computer before he/she become blind and he/she doesn't use screen readers to a fine level as she/he can see the screen to some extent. He/she found it somewhat difficult to identify the proper sound positioning.


Another important finding is that the difficulty of distinguishing different locations in the vertical direction. Most of the users said that it is not that easy to identify differences in the vertical direction (up & down) compared to the horizontal direction (left & right). But some of the users stated that when they get used with the system it become more familiar.


To sum-up, it was a successful user evaluation. This was our first experience in this kind of a evaluation - a human evaluation. It was a wonderful experience for us as well. We are planning to continue this session for another five participants on tomorrow (29th April 2011). Hope it will be a successful one as well!

No comments:

Post a Comment