NEObanking

Combining conversational user interfaces with mobile banking.

Strategies:

Research, UX Design, Prototyping

Tools:

Sketch, Swift, IBM Watson Assistant Service

Time Period:

March – September 2020

Overview Research Insights Ideation Prototype Validation Evolve Learnings

Overview

For my thesis for the bachelor's degree "User Experience Design" (B. Sc.) I conducted independent research on conversational user interfaces, particularly in the field of mobile banking. The results of this 5-month study yielded valuable insights concerning the design of CUIs in an environment that deals with sensitive data but also new opportunities for future work in this context.

This case study breaks down months-long research and is rather in-depth to provide context. For TL;DR, jump to the Evolve or Learnings sections to get a good overview of the results.

Problem

Conversational user interfaces (CUI) have recently gained in popularity, like chatbot systems for customer service or as intelligent assistants on smartphones or other IoT devices. With conversational interfaces, spoken or written language is used to execute tasks instead of clicking several buttons on a graphical interface, which creates an additional barrier between the system and the user.

However, there are often inconsistencies between users’ expectations regarding the capabilities of CUIs and their real skills, which causes frustration and disappointment. Taking into account the current trend of applying CUIs, users will presumably encounter them more frequently. Yet, it seems like there is room for improvement concerning the design of these interfaces to provide a successful conversational user experience. These findings lead to the following research question for my thesis:

How might we enhance both user acceptance and user experience of conversational user interfaces?

How to tackle this question?

Design thinking process used to tackle the research question

My process corresponded with the design thinking approach. First, I conducted interviews with users to collect qualitative insights. Based on the gathered findings, I chose a potential use case for CUIs to sharpen the focus of my thesis. The following extensive literature review provided strategies and best practices for designing CUIs for the selected use scenario. Drawing on these and the insights from the preliminary study, I implemented three different prototypes for conversational mobile banking. I tested them in a user study in which each user performed three tasks and experienced every prototype. After each trial, the participants filled out four questionnaires with a semi-structured interview rounding off the experiment. Afterward, I performed a statistical evaluation of the collected data.

Research

Gathering Insights

To identify and understand the users’ pain-points and needs concerning the interaction with conversational interfaces, I undertook semi-structured interviews with six users who either had experience with conversational assistants in the past or are currently using conversational user interfaces of any kind.

"In my opinion, chatbots are not the solution for everything and have to be used purposefully, but if they are used purposefully and consciously, then they are extremely awesome"

– statement from a participant of the preliminary study

I also gathered the participants’ opinions about current applications of chatbots and what desires and rejections they have about these use cases. To conclude the interviews, users were presented with 15 potential use cases for conversational user interfaces. For each topic, they were asked to rate their willingness to use such conversational agents from 1 (very unlikely) to 7 (very likely) and explain their decision in more detail.

Insights

User Needs and Pain Points

Affinity mapping resulted in +15 themes, some of which were divided into even more refined sub-themes.

The interviews indicated that users currently utilize CUIs for rather simple tasks. The reason for this is that they plainly cannot imagine conversational interfaces for particular services, as they may not have used them yet. Still, the discussions about various use cases of chatbots have shown that a general interest in CUIs and their new areas of application exists. This became clear when existing solutions are complex or when using a conversational approach alleviates and supports users.

Themes found via thematic analysis

To keep things short, I am elaborating on the topic of trust in particular. The respondents cited they have a low level of trust in the competence of CUIs. They are skeptical about the feasibility and convenience of different chatbot applications. On the other hand, the confidence among subjects in conversational interfaces is limited due to the perceived risk they experience when using them. Likewise, users wish that security standards are met by conversational assistants, especially concerning the content that is sensitive or involves private data. What surprised me a lot, however, was that people trust in the intelligence of task-oriented dialog systems, which expresses in the form of usefulness and ease of use.

All mean values for specific CUI use cases
All mean values for the presented 15 potential use cases for conversational interfaces.

In the next step, I chose the use scenario I wanted to focus on in my thesis. Even though conversational interfaces for financial services achieved an average mean value of 3.33, the lack of trust arose in the conversations quite often. From a UX perspective, in particular, the aspect of non-trust is quite intriguing and provides a relevant area of interest.

"If it is a small amount, then I would certainly use such a chatbot. When it comes to financial matters, I would rather confirm this myself by using a button or something like that"

– participant 04

"Especially when it is a financial matter, safety is extremely important. If that can be guaranteed, then [I think it is] great. But I don’t know how this trust can be evoked"

– participant 03

What is Conversational Banking?

The latest development in digital banking is conversational banking, which allows customers to access financial services in the form of natural language via social media and messaging apps as well as voice-based assistants. Thus, financial institutions have a cost-effective opportunity to deliver a personalized digital experience to their clients.

Especially the younger generation is not satisfied with the services offered by banks. They are used to interacting with applications that are personalized and flexibly adapt to their own needs and expect the same from banking solutions. This mindset is not only restricted to young people. Also older customers are not fully satisfied with their current mobile banking solution.

Fintech companies pose another risk as they provide IT solutions that are innovative, customized, and more user-friendly than those currently available from traditional banking institutions. Conversational interfaces for mobile banking can provide added value for users and can potentially help banks overcome technological but also socio-economic challenges.

Ideation

Crafting the Conversation

I implemented three predefined tasks that are handled equally by all CUIs to ensure good comparability of the collected data. The use cases are based on frequently used functions of current banking apps and differ in their degree of complexity.

  1. UC01 - Retrieving account information:
    Task: Check current account balance and list the last transactions.
    No additional TAN is required after login, as banks are allowed to request a TAN every 90 days.
    Complexity: Easy
  2. UC02 - Sending money:
    Task: Send 20 Euros to a saved contact whose name is provided.
    No additional TAN is requested, as contacts and their IBAN can be saved. Thus, a payment without a TAN is permitted.
    Complexity: Medium
  3. UC03 - Pay an invoice:
    Task: Pay an attached invoice containing all payment details such as recipient, amount, IBAN, and booking reference.
    The amount exceeds 30 Euros and the IBAN is not saved, thus a TAN is required. To generate the TAN, the second factor of the SCA is required, which requests the users to do a finger scan.
    Complexity: High

In contrast to graphical interfaces, layout or UI elements were not in the center of my attention since texts are the interface in this type of interaction. I first outlined the flow of the conversation based on three use cases that users perform with current mobile banking solutions. By visualizing the conversation, I was able to quickly determine if the task flow worked and what terms people would use to perform the tasks.

First draft of the conversation

The feedback from the users was mainly related to the assistant’s choice of words, which led to ambiguities for some of them. The conversational flow was also adapted. In the previous version, the assistant's first message asked for the recipient and amount at once when executing the use case of transferring an invoice. However, this was partly overlooked by users and they only answered one of the inquiries. Accordingly, I integrated the user feedback into the design of the subsequent final prototypes.

Meet NEO

Virtual assistant NEO that supports users doing their financial services

After adjusting the task flow, I refined the design of the conversation.

  1. Anthropomorphic features
    Human-like cues are often used to help people anticipate and comprehend the actions and behavior of CUIs. However, this can lead to users overestimating the capabilities of such interfaces, which fail to meet the expectations. Hence users are frustrated and dissatisfied. Thus the virtual assistant NEO is characterized by its robotic look to avoid a too human-like appearance. Still, it shows social mannerisms like saying thank you and apologizing if making mistakes.
  2. Tone of the conversation
    The nature of the conversation between the assistant and user is goal-oriented in the context of financial services. However, to address user feedback that interactions with CUIs are often strict, I focused on the user and assistant working together to achieve a goal. To achieve this, NEO asks clear questions to move the conversation closer to the user's goal, but is also flexible enough in responding to the user's answers.

Prototype

Exploring Conversational Banking

To keep the prototypes as realistic as possible and to address the previously identified security concerns of users, I integrated the elements of the PSD2 policy into the design of the CUIs. This guideline provides for the implementation of a so-called Strong Customer Authentication (SCA). This means that a two-factor authentication (2FA) is used to increase customer protection when performing online payments. This must include at least two factors from the following three categories:

  • Knowledge: something the user knows, e.g. password or pin
  • Ownership: something that is owned by the user, e.g. smartphone or smartcard
  • Inherence: something that can only be assigned to the user, e.g. fingerprint or face recognition

General Features

All CUIs cover the three use cases from above and the assistant NEO always acts the same way. Likewise, only the user actively carries out the sending or transferring of money, but the implementation in the individual interfaces differs. Thus the control remains with the user, which was deemed as essential by the users in the preliminary study concerning financial services. Besides, all CUIs send feedback to the user whether an action has been carried out successfully with the related details.

General features every CUI has implemented
Login screen of ‘NEObanking’ and prompt to use Touch-ID for generating the required TAN.

Speech-based CUI

The CUI communicates its capabilities and the virtual assistant to the user with a welcome screen after the login. The speech variant only allows input via recording voice command, which users can start and stop by themselves. To avoid sending a misunderstood input to NEO, the user can always start a new voice recording which automatically deletes the previous one. Sensitive data, such as the account balance or transaction details, are only displayed visually and not read aloud. Thus the social embarrassment can be counteracted, and also the concerns from the preliminary study that people in the vicinity may overhear financial data have been taken into account.

This is the voice-based CUI
Screenshots of the speech-based CUI. Left: the home screen displaying the virtual assistant and its features. Center: displaying the interaction with the active record button. Right: visual summary of all gathered details by the assistant and confirmation button to execute the transaction.

Hybrid CUI

After the login the users are in the home menu where they can access the different functions of the assistant via buttons, which also provide a brief description of them. If the wrong function was selected, the user can return to the main menu by clicking the back button. This ensures that users always have access to the CA’s capabilities and their descriptions. It also enables users to terminate or restart the interaction. A fixed UI element is present in the chat, that displays details according to the use case. Thus the context of the dialog is more obvious for users and provides an increased transparency of the system.

This is the hybrid CUI
Screenshots of the hybrid CUI. Left: the start screen stating all available functions and their descriptions. Center: displaying the interaction with the fixed context window. Right: displaying all gathered details by the assistant and activated button for the user to confirm the transaction.

Text-based CUI

The two prototypes above were compared in a study with this pure text-based version as baseline. It represents a solely text-based chatbot that is frequently found on messaging platforms. Before an amount, whether a TAN is required or not, is transferred, the conversational agent shows all the payment details entered and asks for confirmation. The user then initiates the final payment with a text-based confirmation.

This is the text-based baseline CUI
Screenshots of the baseline CUI. Left: the start message stating all available functions. Center: displaying the interaction. Right: displaying all gathered details by the assistant and text-based confirmation by the user to confirm the transaction.

The following video shows the implemented use cases with the three different prototypes in action.

Validation

Testing the Prototypes

In a user study, 18 participants tested the three prototypes in a pseudo-randomized task order. Thus, each user completed the three tasks with different CUIs which reduces a carry-over effect. The study included several questionnaires, some of which were adapted to the context: technology acceptance model (Davis), trust (Jian, Bisantz, and Drury), UEQ-s (Schrepp, Hinderks, and Thomaschewski), and perceived security (adapted from Cheng, Lam, and Yeung). The users' ranking of the prototypes and short interviews concluded the user study.

This the text-based baseline CUI

Based on the literature review and the prototypes developed, I formulated the following three hypotheses:

H1: Acceptance differs significantly between the three conversational banking prototypes.

H2: Trust differs significantly between the three conversational banking prototypes.

H3: User experience differs significantly between the three conversational banking prototypes.

Results

The results from the statistical evaluation suggest no significant differences between the prototypes, hence all three hypotheses cannot be accepted. Still, design implications for conversational interfaces for mobile banking from the collected quantitative and qualitative data can be concluded.

Acceptance
In the concluding interviews, 11 of the 18 candidates stated that they are likely to use a conversational interface for financial services. They reported that the prototypes were "easy to use", were "supportive", and were "clearer" or "easier to use" than current solutions. This type of interaction was described by one individual as very "user-friendly", which is also "responsive to personal needs" and "caters to the personal needs of each user individually". Five users are not entirely disinclined to the idea of using conversational interfaces for banking. The participants can imagine such a conversational approach as an add-on, which is integrated into an existing graphical app. Other users mentioned that they would prefer to test this conversational interface for a while to assess whether this type of interaction leads to more efficient banking than existing solutions offer. However, two subjects clearly stated that they would not use conversational solutions for mobile banking.

Trust
Among all interfaces, the speech-based CUI achieved higher mean values than the hybrid CUI did. I was quite suprised by these results, as previous studies and my preliminary study mentioned concerns and limited trust in speech-based interfaces. However, trust is a complex, multidimensional construct. Hence I conclude that the short and one-time interactions users had with all the prototypes were not sufficient enough to draw meaningful conclusions about what design features enhance trust in conversational banking. Accordingly, I conclude that in the context of conversational banking, users’ trust in the CUI is not only influenced by technical or design aspects, like explicitly displaying its capabilities. This is supported by the statement from a user who stated that "concerning financial data, security is more important than speed". Thus implementing security criteria and clearly communicating in which cases those safety prompts are needed might have a considerable impact on trust.

User Experience
The results show that the voice-based CUI achieved the highest average of all three prototypes for both pragmatic (PQ) and hedonic quality (HQ). The high mean value for PQ was somewhat unexpected for me, as subjects noted quite a few issues about the speech-based interaction. Two volunteers said they are generally disinclined towards voice input. Other concerns mentioned were possible errors when entering data, such as transposed digits and issues with speech recognition. Still, many participants mentioned that the voice-based interaction was "efficient", "pleasant", "easy to use", "fast" and "enjoyable". Thus, I assume that despite the concerns mentioned above, the benefits have been more strongly considered in the evaluation of pragmatic quality.

Perceived Security
The interviews indicate that for some people the design of the CUI does not affect its perceived security. Rather, a sense of safety is dependent on the use case currently executed and whether it contained a security query. This becomes apparent as five users explicitly stated that the touch-ID prompt led to a greater feeling of security. Furthermore, I discovered that the favored interface was not necessarily the interface that users perceived to be the most secure one.

Summarising, I conclude that users tend to have a positive attitude towards conversational interface for financial services. I identified a tendency towards the hybrid approach, which should, however, include aspects mentioned in the section Evolve. Apart from design implications that affect the interaction, the safety aspect should not be neglected. Implementing and clearly explaining why specific safety prompts are required to foster users’ acceptance of conversational banking.

Evolve

Uncovering New Opportunities

  1. Support multimodality.
    Users expressed their desire for additional features like using voice or text for interacting with the assistant in the same interface. This allows users to choose how they want to interact with the assistant depending on their surroundings and possible constraints.
  2. Enhance transparency.
    Regarding the feedback offered by the conversational interfaces, users stated that the system’s processes were partly ambiguous, such as whether an amount of money was indeed debited or not. Hence I recommend rendering the processes of the conversational system more transparent for the users.
  3. Provide shortcuts.
    In the case of text-based interfaces, it was argued that this form of input is considered "tedious", "laborious", and "cumbersome". This can be remedied by integrating pre-defined buttons for selecting options or answers, as noted by one subject. Thus, users can reach their goal quicker.

Apart from that, participants also mentioned additional features they desired, like a scan function for capturing invoice details as this is seen as more efficient than typing. Other ideas voiced were a reminder for due invoices and notifications for receipt confirmations.

Learnings

  1. Finding suitable standardized questionnaires to gather quantitative data that also matched the context of my thesis was quite intricate. In my opinion, a balance of quantitative and qualitative metrics is essential for user studies, as they can provide further valuable insights.
  2. I quickly realized that it is essential to write down the intended conversation and evaluate it with users. Hence you can quickly see if the texts are comprehensible and prevent conversational dead ends, and sound natural.
  3. My knowledge of the programming language Swift was limited, and I never used IBM Watson Assistant before my thesis. However, I was able to efficiently implement convincing prototypes with both tools, which probably would not have been possible with conventional prototyping tools.

Next case study:

Personal Project

e-Receipt App