Combining conversational user interfaces with mobile banking.
Research, UX Design, Prototyping
Sketch, Swift, IBM Watson Assistant Service
March – September 2020
For my thesis for the bachelor's degree "User Experience Design" (B. Sc.) I conducted independent research on conversational user interfaces, particularly in the field of mobile banking. The results of this 5-month study yielded valuable insights concerning the design of CUIs in an environment that deals with sensitive data but also new opportunities for future work in this context.
Conversational user interfaces (CUI) have recently gained in popularity, like chatbot systems for customer service or as intelligent assistants on smartphones or other IoT devices. With conversational interfaces, spoken or written language is used to execute tasks instead of clicking several buttons on a graphical interface, which creates an additional barrier between the system and the user.
However, there are often inconsistencies between users’ expectations regarding the capabilities of CUIs and their real skills, which causes frustration and disappointment. Taking into account the current trend of applying CUIs, users will presumably encounter them more frequently. Yet, it seems like there is room for improvement concerning the design of these interfaces to provide a successful conversational user experience. These findings lead to the following research question for my thesis:
How might we enhance both user acceptance and user experience of conversational user interfaces?
My process corresponded with the design thinking approach. First, I conducted interviews with users to collect qualitative insights. Based on the gathered findings, I chose a potential use case for CUIs to sharpen the focus of my thesis. The following extensive literature review provided strategies and best practices for designing CUIs for the selected use scenario. Drawing on these and the insights from the preliminary study, I implemented three different prototypes for conversational mobile banking. I tested them in a user study in which each user performed three tasks and experienced every prototype. After each trial, the participants filled out four questionnaires with a semi-structured interview rounding off the experiment. Afterward, I performed a statistical evaluation of the collected data.
To identify and understand the users’ pain-points and needs concerning the interaction with conversational interfaces, I undertook semi-structured interviews with six users who either had experience with conversational assistants in the past or are currently using conversational user interfaces of any kind.
"In my opinion, chatbots are not the solution for everything and have to be used purposefully, but if they are used purposefully and consciously, then they are extremely awesome"
– statement from a participant of the preliminary study
I also gathered the participants’ opinions about current applications of chatbots and what desires and rejections they have about these use cases. To conclude the interviews, users were presented with 15 potential use cases for conversational user interfaces. For each topic, they were asked to rate their willingness to use such conversational agents from 1 (very unlikely) to 7 (very likely) and explain their decision in more detail.
The interviews indicated that users currently utilize CUIs for rather simple tasks. The reason for this is that they plainly cannot imagine conversational interfaces for particular services, as they may not have used them yet. Still, the discussions about various use cases of chatbots have shown that a general interest in CUIs and their new areas of application exists. This became clear when existing solutions are complex or when using a conversational approach alleviates and supports users.
To keep things short, I am elaborating on the topic of trust in particular. The respondents cited they have a low level of trust in the competence of CUIs. They are skeptical about the feasibility and convenience of different chatbot applications. On the other hand, the confidence among subjects in conversational interfaces is limited due to the perceived risk they experience when using them. Likewise, users wish that security standards are met by conversational assistants, especially concerning the content that is sensitive or involves private data. What surprised me a lot, however, was that people trust in the intelligence of task-oriented dialog systems, which expresses in the form of usefulness and ease of use.
In the next step, I chose the use scenario I wanted to focus on in my thesis. Even though conversational interfaces for financial services achieved an average mean value of 3.33, the lack of trust arose in the conversations quite often. From a UX perspective, in particular, the aspect of non-trust is quite intriguing and provides a relevant area of interest.
"If it is a small amount, then I would certainly use such a chatbot. When it comes to financial matters, I would rather confirm this myself by using a button or something like that"
– participant 04
"Especially when it is a financial matter, safety is extremely important. If that can be guaranteed, then [I think it is] great. But I don’t know how this trust can be evoked"
– participant 03
The latest development in digital banking is conversational banking, which allows customers to access financial services in the form of natural language via social media and messaging apps as well as voice-based assistants. Thus, financial institutions have a cost-effective opportunity to deliver a personalized digital experience to their clients.
Especially the younger generation is not satisfied with the services offered by banks. They are used to interacting with applications that are personalized and flexibly adapt to their own needs and expect the same from banking solutions. This mindset is not only restricted to young people. Also older customers are not fully satisfied with their current mobile banking solution.
Fintech companies pose another risk as they provide IT solutions that are innovative, customized, and more user-friendly than those currently available from traditional banking institutions. Conversational interfaces for mobile banking can provide added value for users and can potentially help banks overcome technological but also socio-economic challenges.
I implemented three predefined tasks that are handled equally by all CUIs to ensure good comparability of the collected data. The use cases are based on frequently used functions of current banking apps and differ in their degree of complexity.
In contrast to graphical interfaces, layout or UI elements were not in the center of my attention since texts are the interface in this type of interaction. I first outlined the flow of the conversation based on three use cases that users perform with current mobile banking solutions. By visualizing the conversation, I was able to quickly determine if the task flow worked and what terms people would use to perform the tasks.
The feedback from the users was mainly related to the assistant’s choice of words, which led to ambiguities for some of them. The conversational flow was also adapted. In the previous version, the assistant's first message asked for the recipient and amount at once when executing the use case of transferring an invoice. However, this was partly overlooked by users and they only answered one of the inquiries. Accordingly, I integrated the user feedback into the design of the subsequent final prototypes.
After adjusting the task flow, I refined the design of the conversation.
To keep the prototypes as realistic as possible and to address the previously identified security concerns of users, I integrated the elements of the PSD2 policy into the design of the CUIs. This guideline provides for the implementation of a so-called Strong Customer Authentication (SCA). This means that a two-factor authentication (2FA) is used to increase customer protection when performing online payments. This must include at least two factors from the following three categories:
All CUIs cover the three use cases from above and the assistant NEO always acts the same way. Likewise, only the user actively carries out the sending or transferring of money, but the implementation in the individual interfaces differs. Thus the control remains with the user, which was deemed as essential by the users in the preliminary study concerning financial services. Besides, all CUIs send feedback to the user whether an action has been carried out successfully with the related details.
The CUI communicates its capabilities and the virtual assistant to the user with a welcome screen after the login. The speech variant only allows input via recording voice command, which users can start and stop by themselves. To avoid sending a misunderstood input to NEO, the user can always start a new voice recording which automatically deletes the previous one. Sensitive data, such as the account balance or transaction details, are only displayed visually and not read aloud. Thus the social embarrassment can be counteracted, and also the concerns from the preliminary study that people in the vicinity may overhear financial data have been taken into account.
After the login the users are in the home menu where they can access the different functions of the assistant via buttons, which also provide a brief description of them. If the wrong function was selected, the user can return to the main menu by clicking the back button. This ensures that users always have access to the CA’s capabilities and their descriptions. It also enables users to terminate or restart the interaction. A fixed UI element is present in the chat, that displays details according to the use case. Thus the context of the dialog is more obvious for users and provides an increased transparency of the system.
The two prototypes above were compared in a study with this pure text-based version as baseline. It represents a solely text-based chatbot that is frequently found on messaging platforms. Before an amount, whether a TAN is required or not, is transferred, the conversational agent shows all the payment details entered and asks for confirmation. The user then initiates the final payment with a text-based confirmation.
The following video shows the implemented use cases with the three different prototypes in action.
In a user study, 18 participants tested the three prototypes in a pseudo-randomized task order. Thus, each user completed the three tasks with different CUIs which reduces a carry-over effect. The study included several questionnaires, some of which were adapted to the context: technology acceptance model (Davis), trust (Jian, Bisantz, and Drury), UEQ-s (Schrepp, Hinderks, and Thomaschewski), and perceived security (adapted from Cheng, Lam, and Yeung). The users' ranking of the prototypes and short interviews concluded the user study.
Based on the literature review and the prototypes developed, I formulated the following three hypotheses:
H1: Acceptance differs significantly between the three conversational banking prototypes.
H2: Trust differs significantly between the three conversational banking prototypes.
H3: User experience differs significantly between the three conversational banking prototypes.
The results from the statistical evaluation suggest no significant differences between the prototypes, hence all three hypotheses cannot be accepted. Still, design implications for conversational interfaces for mobile banking from the collected quantitative and qualitative data can be concluded.
In the concluding interviews, 11 of the 18 candidates stated that they are likely to use a conversational interface for financial services. They reported that the prototypes were "easy to use", were "supportive", and were "clearer" or "easier to use" than current solutions. This type of interaction was described by one individual as very "user-friendly", which is also "responsive to personal needs" and "caters to the personal needs of each user individually". Five users are not entirely disinclined to the idea of using conversational interfaces for banking. The participants can imagine such a conversational approach as an add-on, which is integrated into an existing graphical app. Other users mentioned that they would prefer to test this conversational interface for a while to assess whether this type of interaction leads to more efficient banking than existing solutions offer. However, two subjects clearly stated that they would not use conversational solutions for mobile banking.
Among all interfaces, the speech-based CUI achieved higher mean values than the hybrid CUI did. I was quite suprised by these results, as previous studies and my preliminary study mentioned concerns and limited trust in speech-based interfaces. However, trust is a complex, multidimensional construct. Hence I conclude that the short and one-time interactions users had with all the prototypes were not sufficient enough to draw meaningful conclusions about what design features enhance trust in conversational banking. Accordingly, I conclude that in the context of conversational banking, users’ trust in the CUI is not only influenced by technical or design aspects, like explicitly displaying its capabilities. This is supported by the statement from a user who stated that "concerning financial data, security is more important than speed". Thus implementing security criteria and clearly communicating in which cases those safety prompts are needed might have a considerable impact on trust.
The results show that the voice-based CUI achieved the highest average of all three prototypes for both pragmatic (PQ) and hedonic quality (HQ). The high mean value for PQ was somewhat unexpected for me, as subjects noted quite a few issues about the speech-based interaction. Two volunteers said they are generally disinclined towards voice input. Other concerns mentioned were possible errors when entering data, such as transposed digits and issues with speech recognition. Still, many participants mentioned that the voice-based interaction was "efficient", "pleasant", "easy to use", "fast" and "enjoyable". Thus, I assume that despite the concerns mentioned above, the benefits have been more strongly considered in the evaluation of pragmatic quality.
The interviews indicate that for some people the design of the CUI does not affect its perceived security. Rather, a sense of safety is dependent on the use case currently executed and whether it contained a security query. This becomes apparent as five users explicitly stated that the touch-ID prompt led to a greater feeling of security. Furthermore, I discovered that the favored interface was not necessarily the interface that users perceived to be the most secure one.
Summarising, I conclude that users tend to have a positive attitude towards conversational interface for financial services. I identified a tendency towards the hybrid approach, which should, however, include aspects mentioned in the section Evolve. Apart from design implications that affect the interaction, the safety aspect should not be neglected. Implementing and clearly explaining why specific safety prompts are required to foster users’ acceptance of conversational banking.
Apart from that, participants also mentioned additional features they desired, like a scan function for capturing invoice details as this is seen as more efficient than typing. Other ideas voiced were a reminder for due invoices and notifications for receipt confirmations.