JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

Original Paper

Using the Technology Acceptance Model to Explore User

Experience, Intent to Use, and Use Behavior of a Patient Portal
Among Older Adults With Multiple Chronic Conditions: Descriptive
Qualitative Study

Jennifer Dickman Portz1,2, MSW, PhD; Elizabeth A Bayliss2,3, MD, MPH; Sheana Bull4, PhD; Rebecca S Boxer2,
MD; David B Bekelman1,5, MD; Kathy Gleason2, PhD; Sara Czaja6,7, PhD
1
Division of General Internal Medicine, School of Medicine, University of Colorado, Aurora, CO, United States
2
Institute for Health Research, Kaiser Permanente Colorado, Aurora, CO, United States
3
Department of Family Medicine, School of Medicine, University of Colorado, Aurora, CO, United States
4
mHealth Impact Lab, Colorado School of Public Health, University of Colorado, Aurora, CO, United States
5
Department of Medicine, Eastern Colorado Health Care System, Department of Veterans Affairs, Denver, CO, United States
6
Division of Geriatrics, Weill Cornell Medicine, New York, NY, United States
7
Center for Research and Education on Aging and Technology Enhancement, University of Miami, Miami, FL, United States

Corresponding Author:
Jennifer Dickman Portz, MSW, PhD
Division of General Internal Medicine
School of Medicine
University of Colorado
Mailstop B119
13001 East 17th Place
Aurora, CO, 80045
United States
Phone: 1 303 724 4438
Email: [email protected]

Abstract
Background: Patient portals offer modern digital tools for older adults with multiple chronic conditions (MCC) to engage in
their health management. However, there are barriers to portal adoption among older adults. Understanding portal user interface
and user experience (UI and UX) preferences of older adults with MCC may improve the accessibility, acceptability, and adoption
of patient portals.
Objective: The aim of this study was to use the Technology Acceptance Model (TAM) as a framework for qualitatively describing
the UI and UX, intent to use, and use behaviors among older patients with MCC.
Methods: We carried out a qualitative descriptive study of Kaiser Permanente Colorado’s established patient portal, My Health
Manager. Older patients (N=24; mean 78.41 (SD 5.4) years) with MCC participated in focus groups. Stratified random sampling
was used to maximize age and experience with the portal among participants. The semistructured focus groups used a combination
of discussion and think-aloud strategies. A total of 2 coders led the theoretically driven analysis based on the TAM to determine
themes related to use behavior, portal usefulness and ease of use, and intent to use.
Results: Portal users commonly used email, pharmacy, and lab results sections of the portal. Although, generally, the portal
was seen to be easy to use, simple, and quick, challenges related to log-ins, UI design (color and font), and specific features were
identified. Such challenges inhibited participants’ intent to use the portal entirely or specific features. Participants indicated that
the portal improved patient-provider communication, saved time and money, and provided relevant health information. Participants
intended to use features that were beneficial to their health management and easy to use.
Conclusions: Older adults are interested in using patient portals and are already taking advantage of the features available to
them. We have the opportunity to better engage older adults in portal use but need to pay close attention to key considerations
promoting usefulness and ease of use.

https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 1

(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

(J Med Internet Res 2019;21(4):e11604) doi: 10.2196/11604

KEYWORDS
multiple chronic conditions; personal health record; patient portals; aging; health information technology

usefulness, function, credibility, and satisfaction with the

Introduction technology [24]. There are UI design recommendations for older
Background users [25]; however, little is known about portal use and UX
among older adults with MCC [26]. For example, portal email
Patient portals, also referred to as tethered personal health communication, lab results access, and electronic refill
records, are secure websites for personal health information and capabilities are important features for portal users [18,27], yet,
patient resources directly linked to a provider’s electronic health it is unknown if these tools are commonly used or valued
record [1,2]. Patient portals offer modern digital tools for older specifically by older adults with MCC. As older adults with
adults to engage in health management and with their health MCC have much to gain from using portals, research is needed
care system [3]. In the United States, patient portals vary greatly to better understand use behavior, perceived benefits, and
by provider and health care system, but often provide access to strategies for increasing portal use among this specific
personal health information, email messaging with providers, population [26]. Therefore, the purpose of this study was to
appointment schedulers, and prescription management [1]. qualitatively explore perspectives from older adults with MCC
Patient portals are designed to help patients better manage their regarding Kaiser Permanente Colorado’s (KPCO) established
health with the intention to improve health outcomes, health patient portal, My Health Manager. Framed by the Technology
care communication, and reduce costs [4,5]. With access to lab Acceptance Model (TAM), we qualitatively described the
results and health indicators such as weight, blood pressure, UI/UX, intent to use, and use behaviors among older Kaiser
and cholesterol, patients can promote early intervention when patients with MCC. Our specific research questions included
they encounter a deviation or problem or monitor improvements the following: (1) How do participants use the portal?; (2) Why
if following a new medication, exercise, or diet regimen [6]. do participants use (or not use) the portal as they describe?; (3)
Patients can also access information from various providers and How is the portal useful and usable?; and (4) How do these
necessary medical histories during emergencies to improve care opinions and experiences influence participants’ intent to use
transition coordination [7]. Patient portals can be convenient the portal?
for medication refills, scheduling appointments, and allowing
patients to communicate asynchronously with providers [6,8]. Technology and Acceptance Model
Owing to consumer demand and US government incentives for The TAM is an information technology framework for
health information technology expansion, the adoption and use understanding users’ adoption and use of emerging technologies
of patient portals is on the rise [9]. particularly in the workplace environment and has been tested
Patient portals are a promising but understudied clinical tool in older populations [28,29]. The theory posits that a person’s
particularly in aging populations [10]. As older adults are more intent to use (acceptance of technology) and usage behavior
likely to have multiple chronic conditions (MCC) and higher (actual use) of a technology is predicated by the person’s
health care utilization [11,12], they are likely to benefit from perceptions of the specific technology’s usefulness (benefit from
patient portal use to manage their conditions and health care using the technology) and ease of use. Simply, users are more
services [13,14]. Although older adults are the fastest growing likely to adopt a new technology with high-quality UX design
users of the internet [15,16], a lag in patient portal adoption (ie, usable, useful, desirable, and credible). The TAM also
remains, particularly among the oldest, less affluent, and lower suggests that perceptions of usefulness and ease of use are
educated older adults [17,18]. Although older adults show mediated by external variables including individual differences,
interest in patient portals [19-21], adoption barriers and low system characteristics, social influences, and facilitating
utilization have been identified [22]. Older adults pinpoint conditions.
technology discomfort, privacy and security concerns, and lack Kaiser Permanente Patient Portal: My Health Manager
of relative advantages as primary reasons for not using patient
My Health Manager (Table 1) provides personal health
portals [22,23].
information related to patient diagnosis, prescriptions, laboratory
Understanding older adult opinions about portal user interface results, and vaccination records. To improve provider-patient
(UI) and user experience (UX) may lead to improvements in communication, My Health Manager offers features for patients
the accessibility, acceptability, and adoption of patient portals to email providers and schedule appointments. Health
among older adults with MCC. UI typically focuses on the visual management features that are designed to foster healthy eating
look of the design, including elements related to color, font, and and exercise habits incorporate personalized assessments and
images. UX targets the overall experience related to usability, health self-management tools.

https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 2

(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

Table 1. Patient portal (My Health Manager) features summary.

Feature Function
Appointment Center Patients can schedule or cancel appointments
My Medical Record Patients can view test results, immunization records, medical problem list, and care plans
Pharmacy Center Patients can manage prescriptions and order medications
Health Guides and Health Management Access to health resources and self-management tools for diet, exercise, smoking cessation, and disease
Tools specific care
Message Center Patients can email with their provider
Recently added features e-visit and provider chat functions for nonemergent questions and visits

convenient for the participant. Around 3 to 6 patients

Methods participated in each group. Focus groups were semistructured
This is an exploratory, descriptive qualitative study based on in format, allowing for probing and extended discussion on
data collected from a series of focus groups [30,31]. This method topics of interest to the participants. Before group discussion,
is primarily used to better understand the needs and desired participants were asked to complete a demographics and
outcomes from a particular group of people [32]. As such, this technology utilization survey. The survey collected information
study employed an exploratory, descriptive approach to describe regarding income, education, cell phone, email, internet, digital
UI and UX preferences, intent to use, and use behaviors of communication, and social media use. Other demographic
KPCO’s My Health Manager of older patients with MCC . All variables including race/ethnicity, age, and days since last portal
procedures were approved by the KPCO Institutional Review log-in were captured from KPCO’s electronic medical record
Board. during sampling procedures. Participants were then asked
questions related to My Health Manager. In addition to a
Sample and Recruitment traditional question answer session, patients were asked to think
We identified KPCO patients meeting the following inclusion aloud [33] as a group, whereas the interviewer navigated a mock
criteria: aged ≥65 years, KPCO member for ≥1 year, presence My Health Manager portal (Textboxes 1 and 2). This method
of MCC (Charlson Comorbidity Index >2), and connected to 1 has previously been used to assess health literacy and numeracy
of 3 clinics in the Denver metro area with large geriatric patient of patient portals among patients over 65 years [34]. Focus
populations. Non-English–speaking patients, individuals residing groups were audio-recorded for accuracy in data.
in skilled nursing facilities, and patients with a diagnosis of Data Analysis
dementia were excluded. We then randomly selected potential
participants stratified by age group (65 to 75 years; 76 to 85 A theoretically driven approach [35] based on the TAM was
years; and 86+ years) and portal user status (nonusers and users) used for analysis to capture participants’ opinions and
to ensure participation from older participants and maximize experiences. The analysis was completed by 2 female
the variability of experience with the portal. Nonusers were doctoral-level researchers: JDP, a social work assistant professor
patients not registered for the portal, and users were those with prior qualitative experience, and KG, a research assistant
registered for the portal and logged into the portal within the new to qualitative approaches. Audio files were first
last 6 months. Recruitment letters were mailed to 225 potential professionally transcribed verbatim. The unit of analysis, defined
participants summarizing the study and providing an opt-out as a completion of 1 thought, was determined by the analytic
phone number to call if disinterested. A total of 210 potential team. Units of analysis ranged from a brief 3-word sentence to
participants (n=90 users and n=120 nonusers), who did not a paragraph of 6 sentences and were entered into Microsoft
initially opt-out, were contacted via phone and invited to Excel to ensure that the units were consistent across coders. For
participate in focus groups. Recruitment resulted in an 18% initial coding, coders used a TAM-based theoretically driven
acceptance rate (n=19 users and n=18 nonusers). Of the 37 code book developed a priori to code the units. The codebook
patients that were scheduled to attend focus groups, 24 patients included a list of TAM codes (eg, TAM constructs related to
(n=15 users; n=9 nonusers) participated. We contacted the 13 user intent, usage behavior, usefulness, and ease of use), code
participants who did not show up for their scheduled focus group meaning, and criteria for using each code to capture participants’
to reschedule: 2 patients were unable to reschedule and 11 were perceptions of their intent to use and UX of My Health Manager.
lost to follow-up. To prevent coders from forcing the units to fit with the TAM
framework, coders used a no code option for responses that did
Focus Groups and Question Guide not meet the code meaning and criteria for the a priori codes.
We conducted 6 focus groups (3 nonuser groups and 3 user
groups) lasting approximately 90 min at the KPCO facility most

https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 3

(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

Textbox 1. Overview of focus group questions and portal features for users.
Preliminary questions:

• Why did you enroll in My Health Manager?

• What features do you use most?

Think Aloud Questions:

• As we navigate this feature, what do you think about it?

• Why do you use it?

• How would you improve this feature?

• What helps you use this feature?

• What outcome do you want to achieve by using the feature?

Textbox 2. Overview of focus group questions and portal features for nonusers.
Preliminary questions:

• Are you interested in using My Health Manager?

• Are there reasons why you do not use My Health Manager?

Think Aloud Questions:

• As we navigate the portal, are there features you might like to use?

• What do you like about this feature? Or What do you dislike about this feature?

• Why would you want to use these features? Or Why would you prefer to NOT use these features?

• What supports would you need to use this feature?

• How do you currently accomplish this task (related to feature)?

However, to ensure that participants’ thoughts were fully Technology Acceptance Model Description for My
captured, coders also used a combination of open and in vivo Health Manager
coding (the use of participants’ own words as a code) to add
On the basis of the TAM, Figure 1 illustrates the findings from
inductive codes to the code book as needed. Inter-rater reliability
the focus groups regarding UI and UX, intent to use, and use
was calculated for all transcripts (K=0.98), reflecting adequate
behavior for My Health Manager.
consistency in coding across coders. Patterned coding was then
employed on initial codes to identify (1) patterns in responses Use Behavior
between users and nonusers and (b) patterns in responses Portal users described their use of various My Health Manager
between ease of use, perceived usefulness, intent to use, and features (listed in Table 1). The email Message Center was the
UX. These patterns were then used to form themes related to most popular feature used by My Health Manager users. Of the
each research question and develop overall findings. The participants, 1 stated, “Yeah, I email my doctors a lot!” whereas
analytics team met regularly through the analytic process to another stated, “And I like text chatting with the sending an
discuss codes and correct any disagreements in coding and e-mail to my doctors [feature]; just to ask a question”. The
thematic findings. Pharmacy Center was also commonly used to refill medications.
For example, a participant said, “I use the pharmacy part every
Results time. I hardly ever call in the pharmacy any more”. Viewing
lab results in the My Medical Record page was frequently used.
Participants As 1 participant noted, “I especially like looking up the results
Participants (N=24) were of a mean age of 78 years and were of my test and finding out what those tests are for and if there
primarily white women (Table 2). Patient portal users had [is] anything I need to be concerned about.”
logged in to My Health Manager on an average 17.1 (SD 28.3)
days before recruitment. All but one participant used a cell Participants did not commonly use other My Medical Record
phone regularly, primarily a mobile phone. The majority of features including viewing diagnosis list, care plans, or
participants, regardless of user status, used email and looked immunizations records. In terms of using the Appointment
up information on the Web. Approximately half of the Center to schedule visits, 1 participant explained, “I really like
participants used social media, played video games, and used using the website. I have made appointments and been shocked
video chat, whereas instant messaging was less popular. when I got them the next day on the computer. So sometimes
I try to check there first, and then I’ll call if I feel like I need to

https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 4

(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

come in and I can’t”. However, some participants unsuccessfully neat feature. And then I’ve looked up things for my friends
tried to use the Appointment Center and the majority of when they have questions.” None of the participants had used
participants called the Kaiser phone line to schedule the newly added My Health Manager features including the
appointments. Only 1 participant stated they used the health e-visit or provider chat.
guides and health management tools: “I think that’s a pretty

Table 2. Participant characteristics.

Characteristics Users (N=15) Nonusers (N=9) Total (N=24)
Age (years), mean (SD) 76.4 (4.9) 82.7 (3.4) 78.41 (5.4)
Female, n (%) 12 (80) 5 (56) 17 (71)
White, n (%) 12 (80) 7 (78) 19 (79)
Hispanic, n (%) 1 (7) 2 (11) 3 (13)
Days since My Health Manager log in, mean (SD) 17.1 (28) —a —

Education, n (%)
High School Graduate 1 (7) 5 (56) 6 (25)
Some College Graduate 7 (47) 2 (22) 9 (36)
College Graduate 7 (47) 2 (22) 9 (36)
Income (US$), n (%)
<$30,000 2 (13) 2 (22) 4 (17)
$30,000 to $49,999 7 (47) 6 (67) 13 (54)
$50,000 to $74,999 2 (13) 0 (0) 2 (8)
$75,000 and more 2 (13) 0 (0) 2 (8)
Choose not to answer 2 (13) 1 (11) 3 (13)
Own cell phone, n (%)
Smartphone 12 (80) 5 (56) 17 (71)
A regular or basic phone 12 (80) 3 (33) 6 (25)
Does not have a cell phone 3 (20) 1 (11) 1 (4)
Technology utilization, n (%)
Email 15 (100) 7 (78) 22 (92)
Look up information on the Web 15 (100) 6 (67) 21 (88)
Use social media 8 (53) 5 (56) 13 (54)
Play computer games 12 (80) 3 (33) 15 (63)
Video chat 7 (47) 4 (44) 11 (46)
Instant messaging 6 (40) 2 (22) 8 (33)

a
Not applicable.

https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 5

(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

Figure 1. Technology Acceptance Model description for My Health Manager. UX: user experience; UI: user interface.

that had previously been easy to use. The participant explained,

Perceived Ease of Use “So recently—I don’t know how recently—but Kaiser upgraded,
Although participants stated that the website was “pretty easy and all of a sudden, I couldn’t figure out how to log in. Now
to use,” both portal user and nonuser participants (as shown how stupid is that? So eventually, you know, I scanned and the
during the focus group) were relatively negative about the UI ‘oh, there’s my name in the little arrow,’ but I thought, ‘couldn't
and UX of My Health Manager (Textbox 3). Nonusers quickly they have warned us?’ That was my first thought.”
identified UI design problems related to font size and colors
while viewing the mock portal. In addition to some design Therefore, after an upgrade, users were apprehensive about
issues, portal users noted challenges with using the portal or using the new version and typically found the new version to
follow-up from using the portal system specifically related to be more difficult to use than the previous version.
registering with the system, logging in, and scheduling Computer Self-Efficacy
appointments. Several challenges were related to the back end
Users and nonusers both indicated that they were pretty
of the system. For example, it was easy for patients to order
confident in their ability to “figure things out” on the portal and
their prescriptions on the Web, but in some cases, there were
felt good when they learned how to use a new My Health
problems when participants went to the clinic to pick up the
Manager feature or technology in general. However, participants
prescription. Participants who tried to use the appointment center
discussed that although confident in their ability to use the
said that they could not figure out how to schedule a visit on
portal, the process of learning how to use the website was
the portal. Those that were able to schedule an appointment on
challenging. A user stated:
the portal experienced back-end problems at the clinic when
they arrived to check-in for the visit. Despite some challenges Me and computers have problems anyway. It's like,
in using the portal, many users thought My Health Manager ugh. Probably just inexperience of using the
was easy to use, and in some cases, My Health Manager was computer. I mean, I use the things that they have here,
easier to use than more traditional services. but it's not anything like going onto the website and
that kind of stuff. But it was probably just not knowing
Perceived ease of use was impacted by participants’ level of
where I was going or what I was doing.
computer anxiety and computer self-efficacy.
Responses suggested that older participants believed they could
Computer Anxiety use My Health Manager but acknowledged there would be a
Both portal users and nonusers expressed issues of computer steep learning curve. Although learning to use the portal
anxiety, defined as apprehension or even fear of using the presented difficulties, some participants explained they simply
technology, which led to specific difficulties in using My Health did not have the ability to use specific features. One nonuser,
Manager. As 1 user stated, “Well, I have just a couple of general who was interested in learning more about the provider email
thoughts about the Kaiser website. One is that what we all hate function, stated they could not use the new provider chat
is instant upgrades, right? You hear the word ‘upgrade’ and you function because they could not type or respond quickly enough
shudder.” Users worried that with every update, at some point to instant messages with a provider. They explained, “You have
they would be unable to use My Health Manager and that each to type in your chat...It’s fast...So email would be better.”
new upgraded version created new use challenges. For example,
Perceived Usefulness
during an upgrade, the log in page was modified, making it
difficult for users to find the username and password text box Participants described benefits and drawbacks regarding My
Health Manager (Textbox 4).
https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 6
(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

Textbox 3. Perceived ease of use of My Health Manager.

Participant quotes:

• Barriers

• User interface design

It’s hard for me to see.

The big print I can read. But I have to strain to see the small prints.
It’s like it’s not dark enough. Is that what other people say, too?
• Logging-in

My frustration is, it does not matter what I try to use, it never works. I can do this, this, this, this, this just exactly the
way I remember the way I’m supposed to do it, and then you get down and it does not work. Enter password, wrong.
This or that, something is wrong.
• Scheduling appointments

Nope, I couldn’t make it work...Because you don’t know what’s available. Does (the doctor) have a slot at 2:00 on
such-and-such a day? They don’t tell you that... Step 7? Why is there seven steps just to make an appointment?
• Back-end errors

I don’t want to use (the Appointment Center) because there’s confusion among the people here for instance. And I’ll
tell you, you make (an appointment), for instance I have sun damage because I'm out in the sun a lot, so I made an
appointment to have it checked. And so that was the appointment. When I got here, to see the doctor, the nurse says,
“Well, you have to see a PA.” I said, “Well, I just made an appointment on the website. I made the appointment, I
wrote it down.”
Well, yeah. They went through a period of time, I think, when they were changing over which that’s really the only
problem I’ve ever had with them. And they were significant because medications disappeared. I mean you order them,
they were there. But other than that, it works very well. They straightened it out I think.
• Facilitators

• Simple, quick, and easy to use

It’s nice to be able to see the results and that stuff because that’s easy. Or if the doctor sends you a message. It’s easy
to pick it up.
I have sent emails to my doc, especially when I don't want to come in and usually I get an answer within a very short
time.
The test results are pretty easy to get. I really like it. I like the fact that I can graph my test results as opposed to just
seeing the numbers.

Textbox 4. Perceived usefulness of My Health Manager.

Participant quotes:

• Benefits of using My Health Manager

• Improves patient-provider communication

I love the fact that I can communicate with the doctor or any of the other doctors. I get complete descriptions on blood
work and what happens with that.
You can as you are typing (via email feature), you can think and maybe, “No, that isn’t really what I want to say”
instead of stumbling around. And you can do it more precise...Yeah, more organized. And then before you send it.
For me, it was an easy way to get non-emergent information to the doctor. For me, that's the easiest part of it is I can
send stuff and they’ll either answer me or give me a call, one of the two.
• Saves time and money

And that makes sense, because all it is, is you don’t have to come in for that visit, which, if it’s difficult for you to get
out, if the weather’s crappy and stuff like that.

https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 7

(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

When you initially send the e-mail to your doctor, sometimes they get back to you and they’ll say, “We’ll have a
conference call. I’ve arranged a conference call to talk to you about it.” And that really saves a lot of time.
And it could be something just little or a prescription change or something to that effect that you really don’t have to
come in and see the doctor about. And it’s more an efficient way of really the whole system working.
• Provides patient health information

I am an advocator for people taking control of their own health care versus relying on – that’s not to say I’m going
to self-medicate or anything. But I believe in being well-informed about my healthcare and presenting options to my
doctors and that sort of thing. So I like to be really informed about what’s going on.
But yeah, they don’t have any trouble because it’s nice that I can get messages from my doctors, telling me where
I’m at. Or if I’ve had a blood test, I know that it’s okay or if it’s not
• Drawbacks of using My Health Manager

• Preference for current process

That’s hard. There’s yes and then no, because I don’t hardly call into the hospital or I know when my appointments
are and when I come, they tell me to call in to get my medicine. I don’t because I live so close, it’s even a little walk
for me to come and pick them up. So, the way I feel now, I can still do things like I’m doing now. I mean you have to
walk four or five blocks to come down here, and then I take what I can do by myself. And so, then sometimes I meet
people here that I know, and for me it’s just like getting out for a little trip.
• Distance and serious illness

I think if I had more in my body or that I had more problems that (using My Health Manager) would be good. But I
am never sick. Of course, you never want something to happen. I don’t go to my doctor real often either
If I lived far, it would work very good for me.

Previous Negative User Experience

Benefits
Once a participant had a negative UI with My Health Manager
Portal users expressed a clear benefit in using My Health or a specific feature, they had little interest in trying again. For
Manager, in general and for very specific features. Generally example, a nonuser who tried to register for My Health Manager
speaking, users expressed that My Health Manager was useful was so frustrated with their registration experience that he/she
in communicating with their provider, accessing health did not want to try again, and gave the following explanation:
information, saving time and money, and addressing health
concerns without a clinic visit. I tried to get on [My Health Manager] several years
ago and that was when they were sending the
Drawbacks password by mail. I lost the password, and I forgot
On the contrary, nonusers stated they preferred to accomplish that you had to have (a password). I forgot all that.
health-related tasks using their current process and indicated I again tried to get on it and didn’t have a password
the portal would be more helpful for particular people: those so I thought, “Well, I’ll just start over again.” It
living at a distance and those with serious illness. Nonusers wouldn’t let me, so I said, “Well, the heck with it then.
preferred to use the telephone or clinic pharmacy for Also, as stated above, users expressed many challenges in using
prescriptions, seeing their provider in person when asking the Appointment Center; therefore, participants showed little
nonemergent questions and calling to make clinical intent to use the Appointment Center until the glitches were
appointments. However, it is important to note that participants fixed.
described alternative benefits for their current methods.
Although portal users thought using My Health Manager for Lack of Interest in New Functions
prescription refills was useful in getting medications, nonusers Nonusers and users alike were not interested in using the
liked going to the pharmacy in person because it got them out recently added features of e-visits and chat functions. Few
of the house and kept them active. Nonusers acknowledged that participants saw the value in using these features, articulating
My Health Manager would be useful for patients far away from comments such as, “But I wouldn’t use it because I don’t see
their providers and sicker patients. Nonusers understood why any need to, personally. I’m not saying other people wouldn’t.”
people would want to use the system. It simply did not seem Nonusers wanted to continue seeing their providers in person
useful to them personally. or talking on the phone, and users wanted to continue using the
portal as is. A participant explained about possibly using the
Intent to Use
e-visit feature:
Participants’ intention to continue using or start using My Health
Manager was influenced by their perceived ease of use and Usually, if I want to see the doctor, I want to see the
perceived usefulness. doctor. And I know what it is why I’m going, and what
it is I want to talk to them about. If it’s just real
https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 8
(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

simple, I can just e-mail him or call him. If I want to to adopt burdensome technologies. Therefore, health systems
see my doctor, I undoubtedly have to make an should obtain ongoing UI and UX feedback from older adults
appointment to see him. And if I want to see my with MCC when developing new tools and updates. The
doctor, I want to go see my doctor. Department of Veteran Affairs implemented an ongoing
feedback strategy that fostered adoption of their patient portal
Lack of Awareness of Functions Available
[39].
Intent to use was also influenced by participants’ awareness of
My Health Manager features and access to help using the In terms of perceived usefulness, participants in this study
website. Most participants did not know about the new features, suggested that patients far away from their providers would
and nonusers did not know about the basic features available particularly benefit from the patient portal. However, older
via My Health Manager in general. As 1 nonuser participant adults in rural areas are less likely to use patient portals [40],
stated, “You can see how I read on the computer, because I’d and internet use is lower among people in rural settings,
never seen that—make a—schedule an appointment. That especially among people with MCC [41]. Low internet and
wouldn’t occur to me”; participants did not know what features health technology use in rural communities is often attributed
were available or how to use them. to limited access and awareness [42]. Recent improvements to
broadband access [43] in rural communities may lead to
Discussion increased portal adoption. However, more research is needed
to determine best strategies for promoting portal engagement
Principal Findings among older adults with MCC living in rural settings.
This study supports the growing literature suggesting many Participants indicated the portal would be helpful for sicker
older patients, including those with MCC, are interested in using patients. Although we did not follow up to acquire a better
and are already using patient portals to help manage their health definition of sicker, older patients with serious illness, owing
[22]. This is also the first study to use the TAM to qualitatively to complex care needs, may benefit from portal use. There is
explore the connections between perceived usefulness, ease of some evidence substantiating increased portal adoption among
use, and intent to use for a patient portal among older patients older adults with worse health status [44]. A few studies indicate
with MCC. that people with cancer have positive perceptions of patient
portals [45,46]. Although older adults with serious illness may
The TAM framework and supporting evidence [17,18] indicates
be a target population for portal adoption, little is known about
several external variables influencing perceived ease of use with
patient portal utility for patients with advanced or serious illness.
patient portals. Our study participants identified only 2 external
variables: computer self-efficacy and anxiety. Specific patient Preference for current methods is also a drawback to perceived
portal user trainings offered in-person and/or on the Web may usefulness and barrier to patients’ intent to use. Participants
help older adults learn how to use the portal and when to use value going to the clinic or pharmacy for physical activity and
specific features [36]. Caregivers and family members are also social engagement. With these values in mind, portal designers
helpful in reducing technology-specific anxiety [22,37], but should consider adding functions that encourage older patients
more research is needed to inform portal design for shared to get out of the house and connect personally with their
access with caregivers [38]. providers. As portals advance, it is important to also respect the
patients’ need for a face-to-face connection with their providers.
Email, pharmacy, and medical lab result sections were popular
However, providers may consider using some face-to-face and
and perceived as both useful and easy to use. This use behavior
phone-based time to encourage portal use [21]. Face-to-face
is consistent with other patient portal research in older
and phone-based encouragement (eg, “Did you know you can
populations [4,8]. These features are simple and quick while
schedule your next appointment in the convenience of your
improving perceived patient-provider communication,
home online? Just go to My Health Manager”) from providers
satisfaction with access to health information, and fast
and staff may increase patient awareness of beneficial features.
medication management. Nonusers interested in the portal may
In this study, health management tools and newer features were
be directed to these most popular, usable features. Research
not used primarily owing to a lack of awareness.
shows that once older adults are engaged in a technology, they
tend to be high utilizers [23]. Therefore, promoting adoption of In this sample, patient portals are not preferred by everyone,
popular, easy-to-use features may foster patient satisfaction and and other older adults with MCC may feel similarly. Usage
further use of additional portal features. For example, promoting varies greatly: some patients will never use the portal, other
the email feature initially to encourage a patient to then try the current users will continue to use only a few features, whereas
portal pharmacy system. another group will use every available option. Explicit nonusers
appear to prefer human and face-to-face contact, which has
Other features, particularly the Appointment Center, are difficult
previously been reported from a diverse sample of Kaiser
to use and do not offer perceived benefit to patients in this study.
patients [47]. Regardless of preference, technology-based health
It is easier for patients to simply call to schedule appointments.
care interactions are increasing, and portal use may be expected.
There are also UI design issues related to small fonts and poor
Addressing UI and UX challenges and promoting perceived
coloring, and negative UX influenced participants’ intent to use
benefits (improving commination, saving time, and access to
the portal. These results align with the TAM and previous work
personal information) may improve the intent to use patient
suggesting that technology acceptance is determined by the
portals among older adults with MCC.
perceived value and degree of burden. Older adults are unlikely
https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 9
(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

Limitations relatively well educated, middle income, and technology users,

Although our study is an in-depth analysis of perceptions of lacking specific input from underprivileged populations with
older adults with MCC of a specific patient portal, the use less access to technology resources.
behaviors and experience may differ across portal systems. Conclusions
While employing a descriptive qualitative approach for
Older adults are interested in using patient portals and are
understanding portal UI and UX, intent to use, and use behavior
already taking advantage of the features available to them. We
of My Health Manager, we used only focus groups for data
have the opportunity to better engage older adults to use portals
collection. This work would have benefited from inclusion of
but need to pay close attention to key considerations promoting
other data sources such as observation. We were unable to
usefulness and ease of use. We recommend implementing portal
recruit as many portal nonusers resulting in limitations of our
user trainings, family and caregiver support, ongoing user
nonuser feedback. Although we recruited participants
feedback, and provider encouragement to improve intent to use
representing a wide age range, we did not maintain equal
and adoption among older adults with MCC.
participation from each age group. The sample was also

Acknowledgments
The authors thank Dr Ted Palen for assistance with My Health Manager. This research was supported by funding from the National
Institute on Aging (5T32AG044296). Dr Portz was supported by a career development award funded by the National Institute
on Aging (K76AG059934).

Conflicts of Interest
None declared.

References
1. ONC: Office of the National Coordinator for Health Information Technology. What is a patient portal? URL: https://www.
healthit.gov/faq/what-patient-portal [accessed 2018-12-18] [WebCite Cache ID 74lVGyvCj]
2. ONC: Office of the National Coordinator for Health Information Technology. Are there difference types of personal health
records (PHRs)? URL: https://www.healthit.gov/faq/are-there-different-types-personal-health-records-phrs [accessed
2018-12-18] [WebCite Cache ID 74lVwoHMC]
3. Coughlin SS, Prochaska JJ, Williams LB, Besenyi GM, Heboyan V, Goggans DS, et al. Patient web portals, disease
management, and primary prevention. Risk Manag Healthc Policy 2017;10:33-40 [FREE Full text] [doi:
10.2147/RMHP.S130431] [Medline: 28435342]
4. Ford EW, Hesse BW, Huerta TR. Personal health record use in the United States: forecasting future adoption levels. J Med
Internet Res 2016 Mar 30;18(3):e73 [FREE Full text] [doi: 10.2196/jmir.4973] [Medline: 27030105]
5. Tenforde M, Jain A, Hickner J. The value of personal health records for chronic disease management: what do we know?
Fam Med 2011 May;43(5):351-354 [FREE Full text] [Medline: 21557106]
6. Kruse CS, Argueta DA, Lopez L, Nair A. Patient and provider attitudes toward the use of patient portals for the management
of chronic disease: a systematic review. J Med Internet Res 2015;17(2):e40 [FREE Full text] [doi: 10.2196/jmir.3703]
[Medline: 25707035]
7. Bouri N, Ravi S. Going mobile: how mobile personal health records can improve health care during emergencies. JMIR
Mhealth Uhealth 2014;2(1):e8 [FREE Full text] [doi: 10.2196/mhealth.3017] [Medline: 25098942]
8. Zarcadoolas C, Vaughon WL, Czaja SJ, Levy J, Rockoff ML. Consumers' perceptions of patient-accessible electronic
medical records. J Med Internet Res 2013;15(8):e168 [FREE Full text] [doi: 10.2196/jmir.2507] [Medline: 23978618]
9. Frost and Sullivan. Trade NAVI. 2018. US Patient Portal Market for Hospitals and Physicians Overview and Outlook,
2012-2017 URL: http://tradenavi.net/CmsWeb/resource/attach/report/
[688]U-S-%20Patient%20Portal%20Market%20for%20Hospitals%20and%20Physicians-pdf.pdf [accessed 2018-12-01]
[WebCite Cache ID 74lYjIRuc]
10. Ammenwerth E, Schnell-Inderst P, Hoerbst A. The impact of electronic patient portals on patient care: a systematic review
of controlled trials. J Med Internet Res 2012;14(6):e162 [FREE Full text] [doi: 10.2196/jmir.2238] [Medline: 23183044]
11. Centers for Disease Control and Prevention. 2018. Healthy Aging - State of Aging and Health in America Report Internet
URL: https://www.cdc.gov/aging/pdf/State-Aging-Health-in-America-2013.pdf [accessed 2018-07-12] [WebCite Cache
ID 70rlwMs1W]
12. Federal Interagency Forum on Aging Related Statistics. Washington, DC; 2016. Older Americans 2016: Key Indicators of
Well-Being Internet URL: https://agingstats.gov/docs/LatestReport/Older-Americans-2016-Key-Indicators-of-WellBeing.
pdf[WebCite Cache ID 70rmCzXEh]
13. Gee PM, Greenwood DA, Paterniti DA, Ward D, Miller LM. The eHealth Enhanced Chronic Care Model: a theory derivation
approach. J Med Internet Res 2015;17(4):e86 [FREE Full text] [doi: 10.2196/jmir.4067] [Medline: 25842005]

https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 10

(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

14. Smith SG, Pandit A, Rush SR, Wolf MS, Simon C. The association between patient activation and accessing online health
information: results from a national survey of US adults. Health Expect 2015 Dec;18(6):3262-3273. [doi: 10.1111/hex.12316]
[Medline: 25475371]
15. Pew Research Center. 2018. Mobile Fact Sheet URL: http://www.pewinternet.org/fact-sheets/mobile-technology-fact-sheet/
[WebCite Cache ID 74nPx078E]
16. Pew Research Center. 2018. Tech Adoption Climbs Among Older Adults Internet URL: http://assets.pewresearch.org/
wp-content/uploads/sites/14/2017/05/16170850/PI_2017.05.17_Older-Americans-Tech_FINAL.pdf [accessed 2018-07-13]
[WebCite Cache ID 70tI2l9Ir]
17. Smith SG, O'Conor R, Aitken W, Curtis LM, Wolf MS, Goel MS. Disparities in registration and use of an online patient
portal among older adults: findings from the LitCog cohort. J Am Med Inform Assoc 2015 Apr 25;22(4):888-895. [doi:
10.1093/jamia/ocv025] [Medline: 25914099]
18. Gordon NP, Hornbrook MC. Differences in access to and preferences for using patient portals and other eHealth technologies
based on race, ethnicity, and age: a database and survey study of seniors in a large health plan. J Med Internet Res 2016
Mar 04;18(3):e50 [FREE Full text] [doi: 10.2196/jmir.5105] [Medline: 26944212]
19. Price MM, Pak R, Müller H, Stronge A. Older adults’ perceptions of usefulness of personal health records. Univ Access
Inf Soc 2012 Mar 8;12(2):191-204. [doi: 10.1007/s10209-012-0275-y]
20. Nahm E, Sagherian K, Zhu S. Use of patient portals in older adults: a comparison of three samples. Stud Health Technol
Inform 2016;225:354-358. [Medline: 27332221]
21. Irizarry T, Shoemake J, Nilsen ML, Czaja S, Beach S, DeVito DA. Patient portals as a tool for health care engagement: a
mixed-method study of older adults with varying levels of health literacy and prior patient portal use. J Med Internet Res
2017 Mar 30;19(3):e99 [FREE Full text] [doi: 10.2196/jmir.7099] [Medline: 28360022]
22. Sakaguchi-Tang DK, Bosold AL, Choi YK, Turner AM. Patient portal use and experience among older adults: systematic
review. JMIR Med Inform 2017 Oct 16;5(4):e38 [FREE Full text] [doi: 10.2196/medinform.8092] [Medline: 29038093]
23. Wildenbos GA, Peute L, Jaspers M. Facilitators and barriers of electronic health record patient portal adoption by older
adults: a literature study. Stud Health Technol Inform 2017;235:308-312. [doi: 10.3233/978-1-61499-753-5-308] [Medline:
28423804]
24. Canziba E. Hands-On UX Design for Developers: Design, Prototype, and Implement Compelling User Experiences from
Scratch. Birmingham, UK: Packt Publishing Ltd; 2018.
25. Fisk A, Rogers W, Charness N, Czaja S, Sharit J. Designing for Older Adults: Principles and Creative Human Factors
Approaches, Second Edition. Boca Raton: CRC Press; 2009.
26. Greenberg AJ, Falisi AL, Finney RL, Chou WS, Patel V, Moser RP, et al. Access to electronic personal health records
among patients with multiple chronic conditions: a secondary data analysis. J Med Internet Res 2017 Jun 2;19(6):e188
[FREE Full text] [doi: 10.2196/jmir.7417] [Medline: 28576755]
27. Redelmeier DA, Kraus NC. Patterns in patient access and utilization of online medical records: analysis of MyChart. J Med
Internet Res 2018 Feb 6;20(2):e43 [FREE Full text] [doi: 10.2196/jmir.8372] [Medline: 29410386]
28. Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology: a comparison of two theoretical models.
Manage Sci 1989 Aug;35(8):982-1003. [doi: 10.1287/mnsc.35.8.982]
29. Venkatesh V, Davis FD. A theoretical extension of the technology acceptance model: Four longitudinal field studies.
Management Science 2000 Feb;46(2):186-204. [doi: 10.1287/mnsc.46.2.186.11926]
30. Groves S, Burns N, Jennifer G. The Practice Of Nursing Research: Appraisal, Synthesis, And Generation Of Evidence. St
Louis, Missouri: Elsevier Saunders; 2019.
31. Nayar S, Stanley M. Qualitative Research Methodologies for Occupational Science and Therapy. London, UK: Routledge;
2016.
32. Bradshaw C, Atkinson S, Doody O. Employing a qualitative description approach in health care research. Glob Qual Nurs
Res 2017;4:2333393617742282 [FREE Full text] [doi: 10.1177/2333393617742282] [Medline: 29204457]
33. Jaspers MW, Steen T, van den Bos C, Geenen M. The think aloud method: a guide to user interface design. Int J Med
Inform 2004 Nov;73(11-12):781-795. [doi: 10.1016/j.ijmedinf.2004.08.003] [Medline: 15491929]
34. Sharit J. The roles of health literacy, numeracy, and graph literacy on the usability of the VA's personal health record by
veterans. J Usability Stud 2014;9(4):173-193 [FREE Full text]
35. Saldana J. The Coding Manual for Qualitative Researchers. Thousand Oaks, California: Sage Publications; 2009.
36. Sieck CJ, Hefner JL, Schnierle J, Florian H, Agarwal A, Rundell K, et al. he rules of engagement: perspectives on secure
messaging from experienced ambulatory patient portal users. JMIR Med Inform 2017 Jul 4;5(3):e13 [FREE Full text] [doi:
10.2196/medinform.7516] [Medline: 28676467]
37. Sarkar U, Bates DW. Care partners and online patient portals. J Am Med Assoc 2014;311(4):357-358. [doi:
10.1001/jama.2013.285825] [Medline: 24394945]
38. Latulipe C, Quandt SA, Melius KA, Bertoni A, Miller DP, Smith D, et al. Insights into older adult patient concerns around
the caregiver proxy portal use: qualitative interview study. J Med Internet Res 2018 Nov 2;20(11):e10524 [FREE Full text]
[doi: 10.2196/10524] [Medline: 30389654]

https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 11

(page number not for citation purposes)
XSL• FO
RenderX
 JOURNAL OF MEDICAL INTERNET RESEARCH Portz et al

39. Nazi KM, Turvey CL, Klein DM, Hogan TP. A decade of veteran voices: examining patient portal enhancements through
the lens of user-centered design. J Med Internet Res 2018 Jul 10;20(7):e10413 [FREE Full text] [doi: 10.2196/10413]
[Medline: 29991468]
40. Arcury TA, Quandt SA, Sandberg JC, Miller DP, Latulipe C, Leng X, et al. Patient portal utilization among ethnically
diverse low income older adults: observational study. JMIR Med Inform 2017 Nov 14;5(4):e47 [FREE Full text] [doi:
10.2196/medinform.8026] [Medline: 29138129]
41. Wang J, Bennett K, Probst J. Subdividing the digital divide: differences in internet access and use among rural residents
with medical limitations. J Med Internet Res 2011;13(1):e25 [FREE Full text] [doi: 10.2196/jmir.1534] [Medline: 21371989]
42. Greenberg AJ, Haney D, Blake KD, Moser RP, Hesse BW. Differences in access to and use of electronic personal health
information between rural and urban residents in the United States. J Rural Health 2018 Feb;34(Suppl 1):s30-s38. [doi:
10.1111/jrh.12228] [Medline: 28075508]
43. Federal Communications Commission. 2016. 2016 Broadband Progress Report Internet URL: https://www.fcc.gov/
reports-research/reports/broadband-progress-reports/2016-broadband-progress-report [accessed 2018-12-19] [WebCite
Cache ID 74mvnXiHJ]
44. Lober WB, Zierler B, Herbaugh A, Shinstrom SE, Stolyar A, Kim EH, et al. Barriers to the use of a personal health record
by an elderly population. AMIA Annu Symp Proc 2006:514-518 [FREE Full text] [doi: 10.7717/peerj.3268] [Medline:
17238394]
45. Zide M, Caswell K, Peterson E, Aberle DR, Bui AA, Arnold CW. Consumers' patient portal preferences and health literacy:
a survey using crowdsourcing. JMIR Res Protoc 2016 Jun 8;5(2):e104. [doi: 10.2196/resprot.5122] [Medline: 27278634]
46. Alpert JM, Morris BB, Thomson MD, Matin K, Brown RF. Implications of patient portal transparency in oncology:
qualitative interview study on the experiences of patients, oncologists, and medical informaticists. JMIR Cancer 2018 Mar
26;4(1):e5 [FREE Full text] [doi: 10.2196/cancer.8993] [Medline: 29581090]
47. Lyles CR, Allen JY, Poole D, Tieu L, Kanter MH, Garrido T. "I Want to Keep the Personal Relationship With My Doctor":
Understanding Barriers to Portal Use among African Americans and Latinos. J Med Internet Res 2016 Dec 03;18(10):e263
[FREE Full text] [doi: 10.2196/jmir.5910] [Medline: 27697748]

Abbreviations
MCC: multiple chronic conditions
KPCO: Kaiser Permanente Colorado
TAM: Technology Acceptance Model
UI: user interface
UX: user experience

Edited by G Eysenbach; submitted 19.07.18; peer-reviewed by T Irizarry, C Jacob, J Hefner, T Risling; comments to author 08.10.18;
revised version received 21.12.18; accepted 23.01.19; published 08.04.19
Please cite as:
Portz JD, Bayliss EA, Bull S, Boxer RS, Bekelman DB, Gleason K, Czaja S
Using the Technology Acceptance Model to Explore User Experience, Intent to Use, and Use Behavior of a Patient Portal Among
Older Adults With Multiple Chronic Conditions: Descriptive Qualitative Study
J Med Internet Res 2019;21(4):e11604
URL: https://www.jmir.org/2019/4/e11604/
doi: 10.2196/11604
PMID: 30958272

©Jennifer Dickman Portz, Elizabeth A Bayliss, Sheana Bull, Rebecca S Boxer, David B Bekelman, Kathy Gleason, Sara Czaja.
Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 08.04.2019. This is an open-access article
distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which
permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal
of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on
http://www.jmir.org/, as well as this copyright and license information must be included.

https://www.jmir.org/2019/4/e11604/ J Med Internet Res 2019 | vol. 21 | iss. 4 | e11604 | p. 12

(page number not for citation purposes)
XSL• FO
RenderX