¹University of North Dakota, Department of Education, Health, & Behavior Studies, Grand Forks, North Dakota

²University of Wyoming, Wyoming Institute for Disabilities, Laramie, Wyoming

³University of Wyoming, Department of Psychology, Laramie, Wyoming

Received Date: February 13, 2023

Publication Date: March 16, 2023

*Corresponding Author: Ethan Dahl, University of North Dakota, 231 Centennial Dr Stop 7189, Grand Forks, ND 58202-7189. Email: [email protected]

Citation: Dahl E, et al. (2023). COVID-19 and the Transition of Rural Mental Health Providers to Telebehavioral Health. Clin Res. 4(1):4.

ABSTRACT

The present investigation was designed to evaluate mental health practitioners’ rapid transition to telehealth utilization that was precipitated by the SARS-CoV-2 global pandemic (COVID-19). Though prior research has described providers’ adoption of distance technologies to support clinical care and client engagement efforts, these processes are typically gradual and planful. The coronavirus pandemic was comparatively unique in necessitating rapid adoption and utilization of telehealth modalities with variable guidance, formal training, and minimal resources. We surveyed 39 mental health professionals recruited from statewide mental health organizations and networks in Wyoming. Participants completed a survey regarding their telehealth usage and transition as well as measures of their knowledge of established telebehavioral health competencies. Contrary to hypotheses, providers who reported that the pandemic caused them to begin using telehealth in their practice reported greater knowledge of and confidence in telebehavioral health competencies relative to those providers who had already been using telehealth prior to the pandemic. Because those who transitioned to telehealth rapidly did not access greater telehealth training or informational resources and were not more knowledgeable about established telehealth competencies, they may be overestimating their proficiency and preparedness. Implications such as the need for additional training for these providers and concerns regarding the quality of telebehavioral health practice are presented and discussed.

Keywords: Telehealth, Telebehavioral Health Competencies, COVID-19, Provider confidence 

CLINICAL IMPACT STATEMENT

Though telebehavioral health is a particularly advantageous practice innovation that facilitated the delivery of mental healthcare during the recent global pandemic, rapid transition to this modality may have been problematic in some regards. Specifically, the impact of the current study is that providers who were pushed into using telebehavioral health were unaware of the competencies related to the quality provision of this type of care and may have been overestimating their proficiency and preparedness.

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) global pandemic (COVID-19 pandemic) precipitated a sea change in mental healthcare delivery and access (Monaghesh & Hajizadeh, 2020) [1]. Although secure and encrypted videoconferencing-based technologies have long held great promise in facilitating access to psychotherapeutic services, they nevertheless remained underutilized relative to their potential benefits. Certainly, there will be considerable, enduring benefits to the rapid adoption of and transition to telehealth (e.g., Gentry et al., 2021) [2], but comparatively little is known about mental health providers’ experiences in hastily transitioning to this treatment modality. The present study was designed to examine mental health providers’ approach to and experiences with pandemic-necessitated transitions to telehealth. In particular, we were interested in understanding knowledge and confidence in established telebehavioral health competencies as providers quickly navigated this transition in modality and format for providing care.

Telebehavioral health is the use of telecommunications technology to provide healthcare services, specifically for mental health providers. Telebehavioral health is an efficacious form of healthcare delivery – often rivaling outcomes seen in face-to-face clinical trials (e.g., Hilty et al., 2013; Varkder et al., 2019; Acierno et al., 2017) [3-5], however prior use of this modality (pre-COVID-19 pandemic) has typically followed strategic, planful, and gradual implementation. Those utilizing telehealth prior to the COVID-19 pandemic in 2020 were able to consider practice needs, technology platforms, implementation barriers, and solutions, and usually had ample time to seek out information and training; all of which represent principles known to support successful telemedicine implementation (Yellowlees, 2005) [6]. In contrast, the coronavirus pandemic required providers to transition rapidly and on a large scale to distance service delivery models. This allowed clinicians to meet the needs of existing clients and to prevent lags in service implementation at the very moment when anxiety and mood difficulties were being exacerbated by current events (Schroeder et al., 2021) [7].

An example of this rapid adoption can be seen in the frontier state of Wyoming and the Wyoming Telehealth Network (WyTN) which carries a mission to increase access to care and improve health outcomes through professional development, collaboration, and leveraging of telecommunications technology to support healthcare entities, providers, and specialists. The WyTN, which only serves a fraction of the providers in the state, recently noted that the number of telebehavioral health users increased approximately seventeen-fold over the course of the first year of the pandemic – from 138 providers in February 2020 to 2,344 users by March of 2021 (Wyoming Telehealth Network, 2021) [8]. Additionally, in FY 2021 Wyoming Medicaid paid over $8,567,636 in telehealth services for a total of 60,781 claims, over half (32,149) were for telebehavioral health services (Wyoming Medicaid, 2022) [9]. Rural providers were clearly pushed into telebehavioral health by the COVID-19 pandemic. The degree to which service providers sought out and accessed formal training resources, literatures, and opportunities, however, is likely highly variable and idiosyncratic.

Movement to a new and innovative modality of care delivery such as telebehavioral health should be predicated by relevant preparations to ensure efficacious care. Requisite training for effective service delivery typically requires information about legal and structural issues (e.g., HIPAA compliance and confidentiality, platform features, telehealth billing codes, and regulations pertaining to cross-jurisdictional services), as well as pressing clinical concerns and adaptations (e.g., risk and safety assessments and protocols for healthcare delivered at a distance, necessary adaptations to treatment techniques, etc.) and includes a rigorous evaluation of impact and clinical outcomes (e.g. cost sustainability of service delivery, efficient achievement of desired outcomes, etc.) (Yellowlees, 2005) [6]. A steep learning curve may be present for providers engaging with telehealth as a new service provision during ideal times, which certainly was not the case in Spring and Summer of 2020 as providers witnessed an increase in both the need for telehealth services and the overall demand for general mental health services (Conrad et al., 2020) [10]. For many providers, especially rural ones, there was also a lack of training resources available for providers making this transition during the COVID-19 pandemic (Shroeder et al., 2021) [7]. Rural telebehavioral health providers seemingly faced a multitude of barriers in their transition to using telehealth during the COVID-19 pandemic.

Fortunately, resources to guide practice do exist and have been articulated and advanced by the Coalition for Technology in Behavioral Science (CTiBS; Maheu, et. al., 2018) [11]. These include competencies that are intended to provide practices with key knowledge and skills needed to improve the quality of telehealth services, inform training, and are currently the only standards for the provision of healthcare services at a distance in any field. These competencies were created to align with the competencies, guidelines, and position statements from the professions of psychology, social work, marriage and family therapy, psychiatry, and the American Telemedicine Association (Hiltyet al., 2018) [12]. The seven competencies include Clinical Evaluation and Care, Virtual Environment and Telepresence, Technology, Legal and Regulatory Issues, Evidence-Based and Ethical Practice, Mobile Health and Apps, and Telepractice Development. For the current study, we were interested in assessing providers’ confidence, knowledge, and proficiency in these competencies as a function of whether their transition to telehealth was precipitated by the pandemic. We hypothesized that compared to those who had already incorporated telehealth services into their practice prior to the pandemic, the participants who transitioned rapidly to telehealth delivery because of the pandemic would report more limited knowledge and confidence in adherence to CTiBS competencies. If true, it would suggest that the rapidity of the necessary transition may have adversely impacted the quality of care and competent delivery of services (e.g., disparities – Chang et al., 2021) [13].

METHOD

The study was approved as exempt by the University of Wyoming Institutional Review Board (Protocol #20200520CH02758, Date-5/20/2020), and informed consent was obtained from all participants. Data were collected during the summer of 2020. Participants (N = 39) for this study included mental health providers primarily licensed to provide services in the state of Wyoming. Providers were invited to participate in the study via email communications and told that the study would focus on COVID-19 and Telebehavioral Health Competencies. Email communications were distributed via the Wyoming Telehealth Network (WyTN) listserv to registered mental health providers as well as email distributions to larger mental and behavioral health organizations such as the National Association of Social Workers (NASW) and American Psychological Association (APA) and through the state of Wyoming Behavioral Health Division and the mental health and substance abuse centers association (WAMHSAC). Recruitment communications for the larger organizations occurred once during the summer of 2020 while recruitment emails from the WyTN occurred 3 times during the recruitment period. The estimation of the response rate for this study is difficult as the count of individuals reached via these methods was unobtainable. It is estimated that there are ~2500 telebehavioral health providers in Wyoming, meaning the response for the current study was ~1% of the total population, which was not fully reached by these recruitment methods. Participants were told that the study would take ~15 minutes of their time and were offered the opportunity to enter gift card drawings for their time. Identifying information was not gathered from participants.

The providers completed a survey via the REDCap web-based data management system. They were informed that the survey would ask them about their use of telebehavioral health. The survey included items asking participants about their current and past use of telehealth (relative to the onset of the covid pandemic), their transition to telehealth, their background as a mental health provider, and their knowledge about using telebehavioral health. All items were developed by the researchers for the purposes of the current study. Participants were specifically asked about their knowledge and confidence regarding using telebehavioral health and the seven competencies of the Coalition for Technology in Behavioral Science (CTiBS) guidelines for telebehavioral health. Participants were presented with the competency name as well as the full description of the competency from CTiBS (see Appendix A). The CTiBS guidelines are one of the few frameworks that exist for telebehavioral health and were developed to improve the quality of care via telebehavioral health and to standardize training (Maheu, Drude, Hertlein, & Hilty, 2018)[13,14]. Participants rated their confidence, knowledge, and proficiency rating in all seven competencies and were also given total scores across all ratings.

RESULTS

Participants

A total of N = 39 providers completed the survey which gathered data from June 3rd 2020 to August 12th 2020. These participants were mostly licensed to provide telebehavioral services in the state of Wyoming (n = 36) but also listed licensing in Rhode Island (n = 4), Massachusetts (n = 2), Michigan (n = 2), Colorado (n = 1), Illinois (n = 1), and Wisconsin (n = 1). Most of the providers were currently providing telebehavioral health services (n = 37, 95%) at the time of the study and most planned to continue providing these services in the future (n = 34, 79%). A majority of the providers were not providing telebehavioral health services prior to the pandemic (n = 26, 67%) and indicated that the pandemic caused them to begin offering telebehavioral health services (n = 27, 69%). The pandemic also caused most of the providers to seek out additional information on providing telebehavioral health services (n = 34, 87%). Provider participants had between 1-42 years of experience as a mental health provider (M = 15.39, SD = 11.90) but only between 0-13 years practicing telebehavioral health (M = 1.89, SD = 3.07). Most providers also reported that their professional training and/or education did not involve experience with using telebehavioral health (n = 31, 82%). The providers averaged 15.88 telebehavioral appointments per week (SD = 12.11) at the time of data collection and averaged 3.09 telebehavioral appointments per week (SD = 6.78) prior to the COVID-19 pandemic. The majority of total services being provided were being delivered via telebehavioral health (M = 69.69%, SD = 40.53%) during the study but responses ranged widely on this item.

Confidence and Knowledge of Telebehavioral Health Competencies

Knowledge regarding using telebehavioral health as well as knowledge and confidence regarding the seven CTiBS competencies were measured across all participants. Participants rated their overall knowledge of using telebehavioral health from 1 (Very Unknowledgeable) to 5 (Very Knowledgeable) with an average score of M = 3.95 (SD = 1.06). Prior to being asked about the specific CTiBS competencies participants indicated their familiarity with the guidelines and most were not aware (n = 34, 87%). In comparing the seven competencies the correlation between knowledge (1 – Not at All – 5 Very) and confidence (1 – Not at All – 5 Very) across all items was very high r(27) = .96, p < .001. Means and standard deviations for each competency are listed in Table 1. Scores were high across all competencies despite most participants reporting not being familiar with the competencies.

Table 1: Means and Standard Deviations of Coalition for Technology in Behavioral Science (CTiBS) Competencies

CTiBS Competency	Confidence	Knowledge
1. Clinical Evaluation & Care	3.65 (0.80)	3.65 (0.80)
2. Virtual Environment & Telepresence	3.97 (0.77)	3.93 (0.74)
3. Technology	3.76 (0.74)	3.79 (0.77)
4. Legal & Regulatory Issues	3.72 (0.78)	3.69 (0.89)
5. Evidence-Based & Ethical Practice	3.83 (0.66)	3.82 (0.67)
6. Mobile Health Technologies Including Applications (Apps)	3.00 (1.00)	2.97 (1.05)
7. Telepractice Development	3.07 (0.96)	3.07 (1.00)

Pandemic-Precipitated Telehealth Practice and Self-Rated Competency

Participant data were split to evaluate those providers that were pushed by the pandemic into practicing telebehavioral health and those that were providing telebehavioral health services prior to the pandemic. Participants were asked to either endorse or not the item: “Did the COVID-19 pandemic cause you to begin using telebehavioral health?” This split led to a sample of n = 27 that indicated “Yes” that the pandemic caused them to begin using telebehavioral health and n = 12 that indicated “No”. MANOVA analyses comparing the two groups on the knowledge and confidence ratings for the seven competencies showed significantly higher ratings for individuals who said the pandemic caused them to start using telebehavioral health on confidence in competency 1, confidence in competency 3, knowledge on competency 3, confidence in competency 5, confidence in competency 6, and knowledge on competency 6 (Table 2).

Table 2: Coalition for Technology in Behavioral Science (CTiBS) Competency Comparisons for Caused/Did Not Cause Groups.

CTiBS Competency	Pandemic Caused	N	Confidence	F	p	η2	Knowledge	F	p	η2
1. Clinical Evaluation & Care	Yes	21	3.86 (0.79)	5.25	0.029	0.153	3.76 (0.77)	1.41	0.244	0.046
	No	10	3.20 (0.63)				3.40 (0.84)
2. Virtual Environment & Telepresence	Yes	20	4.10 (0.64)	1.89	0.181	0.063	4.05 (0.61)	1.52	0.228	0.051
	No	10	3.70 (0.95)				3.70 (0.95)
3. Technology	Yes	19	4.00 (0.67)	7.16	0.012	0.210	4.05 (0.62)	7.68	0.010	0.221
	No	10	3.30 (0.68)				3.30 (0.82)
4. Legal & Regulatory Issues	Yes	19	3.89 (0.66)	2.67	0.114	0.090	3.89 (0.66)	3.15	0.087	0.104
	No	10	3.40 (0.97)				3.30 (1.16)
5. Evidence-Based & Ethical Practice	Yes	19	4.00 (0.58)	4.21	0.050	0.135	3.95 (0.62)	2.18	0.152	0.077
	No	10	3.50 (0.71)				3.56 (0.73)
6. Mobile Health Technologies Including Applications (Apps)	Yes	19	3.26 (1.05)	4.26	0.049	0.136	3.26 (1.05)	5.05	0.033	0.158
	No	10	2.50 (0.71)				2.40 (0.84)
7. Telepractice Development	Yes	19	3.16 (1.02)	0.46	0.502	0.017	3.21 (1.03)	1.11	0.301	0.040
	No	10	2.90 (0.88)				2.80 (0.92)

Participants additionally rated themselves on each of the seven CTiBS competencies as either “Novice”, “Proficient”, or “Authority”. Total counts across the competencies were calculated to determine an overall classification for participants for the purpose of comparisons. When comparing those participants who indicated that the pandemic caused them to start using telehealth and those who indicated that the pandemic did not cause them to start using telehealth, a chi-square test indicated a significant difference between the groups who indicated Novice or Proficient, χ2 (1, N = 31) = 6.05. p = .01 (Authority classifications were excluded due to 0 respondents in the “Did Not Cause” cell). Table 3 shows counts for this comparison. Interestingly, those that said that the pandemic caused them to start using telehealth indicated that they were proficient on the CTiBS competencies much more frequently than those that said the pandemic did not cause them to start using telehealth.

Table 3: Authority Classifications for Coalition for Technology in Behavioral Science (CTiBS) Competencies.

Competency Rating	Total	Pandemic Caused Telebehavioral Health	The Pandemic Did not Cause Telebehavioral Health
Novice	83 (44%)	44 (37%)	39 (56%)
Proficient	105 (56%)	74 (63%)	31 (44%)

The differences between these two groups were also analyzed regarding whether or not the pandemic caused the providers to seek more resources. Across the full sample (N = 39), n = 34 participants indicated that the pandemic caused them to seek out additional information on providing telebehavioral health services with n = 23 coming from the pandemic-caused group and n = 11 coming from the pandemic did not cause group. Participants then listed the places they went to find additional information. Those participants that indicated that the pandemic caused them to begin using telehealth listed an average of 2.3 resources sought out while those that the pandemic did not cause listed an average of 1.89 resources sought. This difference was not significant. Further sub-setting the data into pandemic caused/did not cause and novice/proficient categories showed that pandemic caused/proficient (n =16) averaged 2.5 resources, a pandemic caused/novice (n =4) averaged 1.5 resources, pandemic did not cause/proficient (n =4) averaged 1.75 resources, and pandemic did not cause/novice (n =6) averaged 0.75 resources. These differences were not significant.

DISCUSSION

The current study sought to understand the impact of the COVID-19 pandemic on the telebehavioral health service delivery by Wyoming providers. Given the abrupt and sudden transition to this modality of behavioral health practice, the current study sought to understand the knowledge and confidence of providers to better understand if and how they were prepared for the virtual environment they and their clients were thrust into. While most providers indicated that they were not aware of the CTiBS competencies, responses across all providers indicated relatively high levels of knowledge and confidence regarding the competencies (Table 1). When splitting the providers into those that the COVID-19 pandemic caused to begin using telebehavioral health and those that were using telebehavioral health prior to the pandemic, interesting differences emerged. Contrary to our hypothesis, providers whom the pandemic caused to transition to telebehavioral health were more confident and reported being more knowledgeable regarding CTiBS competencies than those who were using telebehavioral health prior to the COVID-19 pandemic.

Confidence can be an important aspect in success as a telebehavioral health provider (Felker et al., 2021) [15]. However, an overestimation of one’s knowledge and skills can be dangerous (Johnson & Fowler, 2011) [16]. For healthcare and telebehavioral health providers, overconfidence could lead to a host of negative outcomes including complacency and overestimation of diagnostic accuracy (Berner & Graber, 2008), failure to seek additional resources (Meyer et al., 2013; Rosenbloom et al., 2005) [17,18], incorrect management of clients/patients and seeking fewer consultations (Kovacs et al., 2020) [19], and general diagnostic error (Berner & Graber, 2008; Cassam, 2017) [20,21]. While not completely clear from the current data, it appears as though providers who were thrust into using this modality of service delivery brought with them a level of confidence regarding the principles of telebehavioral health as well as a reported knowledge level that exceeded that of those that were using telebehavioral health before the pandemic. Though beyond the scope of the current study, it is interesting to consider if this overconfidence led to the successful provision of care or not. Future longitudinal studies could consider this among telebehavioral health clients and providers.

The providers who reported overconfidence and being more knowledgeable also rated themselves as proficient in the CTiBS principles much more frequently. The danger of this overestimation is well documented as a cognitive bias known as the Dunning-Kruger effect (Dunning, 2011; Kruger & Dunning, 1999) [22,23]. This effect is described as the issue of individuals who do not realize their own incompetence in an area often overestimating their knowledge and skills in that area. The issue is that individuals who do not fully understand something do not possess a proper understanding of how to evaluate performance in that activity. For the providers in this study who had not used telebehavioral health prior to the pandemic, their assessment of their proficiency in these areas was based on a potential lack of knowledge in the first place. They likely did not possess a full understanding of the appropriate skills and strategies necessary to be proficient in the CTiBS principle areas and so rated themselves highly. This is one potential explanation for the current findings and highlights the importance of breaking the initial lack of understanding for these providers.

Though we cannot rule out the possibility that those who transitioned to telehealth rapidly because of the pandemic were more knowledgeable and competent, this interpretation of the data seems unlikely for several reasons. First, the overwhelming majority of all providers surveyed in this investigation were unaware of specific competencies outlined by the CTiBS. Second, providers who began utilizing telebehavioral health modalities abruptly due to the pandemic did not differ from those who had already been using telehealth modalities with respect to the number of informational resources utilized and consulted. Lastly, it is unlikely that one could truly become proficient or expert in any treatment modality in a duration of a few months. The fact that providers who rapidly transitioned to telehealth due to the pandemic were significantly more likely to rate themselves as proficient relative to those who had been using telehealth for years suggests that self-appraisals of knowledge and competency among the former group may well have been inflated. The number of providers who reported being unaware of the CTiBS competencies (87%) is worth noting. These competencies represent one of, if not the only, comprehensive set of principles of telebehavioral health provision but it is alarming to see such a high percentage of providers that were unaware of their existence. There is a clear need for additional education and awareness regarding these competencies among these providers as well as among all telehealth providers in rural areas (Shroeder et al., 2021) [7]. Training programs both during the primary training of these professionals and as part of ongoing professional development should emphasize these principles and areas of competence.

LIMITATIONS

The current research is based on a relatively small sample of telebehavioral health providers in the rural state of Wyoming. The awareness of CTiBS principles and the knowledge and confidence about these principles are likely to be different among different provider populations. While Wyoming would appear to be an exemplary area for telebehavioral health, historical data from the Wyoming Telehealth Network indicates a low rate of adoption of this type of care provision, which likely contributed to the lack of awareness shown in the current data. It is also possible that the respondents in this study possess certain discerning characteristics, given the low response rate for the study, and the results presented here could be due to response bias. The results also may not be generalizable beyond providers from Wyoming due to the concentration of these individuals in the study. Future work should obtain a larger sample whether within a particular state or beyond.

Overall, this study highlights a potentially concerning issue with telebehavioral health providers that transitioned to virtual care provision during the COVID-19 pandemic. These providers reported high levels of confidence and knowledge of key telebehavioral health competencies even when they were not utilizing telebehavioral health prior to the pandemic. Surprisingly, those not using telebehavioral health before the pandemic rated themselves higher on several of the telebehavioral health competencies than those who were using telebehavioral health to provide services before the pandemic, suggesting potential overconfidence of these providers. The current study provides some insight into the transition of telebehavioral health providers to using virtual forms of service provision during the COVID-19 pandemic. More work must be done to ensure that providers are equipped to provide the best care possible in the new mental health landscape of the world.

CONFLICT OF INTEREST

We have no known conflict of interest to disclose.

ACKNOWLEDGEMENTS

The authors would like to acknowledge the Equality State Research Network (ESRN), which is Wyoming’s community and practice-based research network, and the Wyoming Telehealth Network (WyTN) for their support in the implementation of this study and recruitment of research participants.

REFERENCES

1. Monaghesh E, Hajizadeh A. (2020). The role of telehealth during COVID-19 outbreak: a systematic review based on current evidence. BMC Public Health. 20(1):1-9.
2. Gentry MT, Puspitasari AJ, McKean AJ, Williams MD, Breitinger S, Geske JR, et al. (2021). Clinician satisfaction with rapid adoption and implementation of telehealth services during the COVID-19 pandemic. Telemed e-Health. 27(12):1385-1392.
3. Hilty DM, Ferrer DC, Parish MB, Johnston B, Callahan EJ, Yellowlees PM. (2013). The effectiveness of telemental health: a 2013 review. Telemed J E Health. 19(6):444–454.
4. Varker T, Brand RM, Ward J, Terhaag S, Phelps A. (2019). Efficacy of synchronous telepsychology interventions for people with anxiety, depression, posttraumatic stress disorder, and adjustment disorder: A rapid evidence assessment. Psychol Serv. 16(4):621–635.
5. Acierno R, Knapp R, Tuerk P, Gilmore AK, Lejuez C, Ruggiero K, et al. (2017). A non-inferiority trial of prolonged exposure for posttraumatic stress disorder: in person versus home-based telehealth. Behav Res Ther. 89:57-65.
6. Yellowlees PM. (2005). Successfully developing a telemedicine system. J Telemed Telecare. 11(7):331-335.
7. Schroeder S, Roberts H, Heitkamp T, Clarke B, Gotham HJ, Franta E. (2021). Rural mental health care during a global health pandemic: Addressing and supporting the rapid transition to tele-mental health. J Rural Mental Health. 45(1):1-13.
8. Wyoming Medicaid. (2022). Wyoming Medicaid SFY 2021 Annual Report.
9. Wyoming Telehealth Network. (2021). Provider directory. Wyoming Telehealth Network.
10. Conrad R, Rayala H, Diamond R, Busch B, Kramer N. (2020). Expanding telemental health in response to the COVID-19 pandemic. Psychiatric Times.
11. Maheu MM, Drude KP, Hertlein KM, Hilty DM. (2018). A framework of interprofessional telebehavioral health competencies: Implementation and challenges moving forward. Academic Psychiatry. 42(6):825-833.
12. Hilty DM, Maheu MM, Drude KP, Hertlein KM. (2018). The need to implement and evaluate telehealth competency frameworks to ensure quality care across behavioral health professions. Academic Psychiatry. 42(6):818-824.
13. Maheu MM, Drude KP, Hertlein KM, Lipschutz R, Wall K, Hilty DM. (2018). Correction to: an interprofessional framework for telebehavioral health competencies. J Technol Behav Sci. 3(2):108-140.
14. Chang JE, Lai AY, Gupta A, Nguyen AM, Berry CA, Shelley DR. (2021). Rapid transition to telehealth and the digital divide: implications for primary care access and equity in a post‐COVID era. Milbank Quarterly. 99(2):340-368.
15. Felker BL, McGinn MM, Shearer EM, Raza GT, Gold SD, Kim JM, et al. (2021). Implementation of a telemental health training program across a mental health department. Telemedicine Reports. 2(1):26-31.
16. Johnson D, Fowler J. (2011). The evolution of overconfidence. Nature. 477:317–320.
17. Meyer AN, Payne VL, Meeks DW, Rao R, Singh H. (2013). Physicians’ diagnostic accuracy, confidence, and resource requests: a vignette study. JAMA. 173(21):1952-1958.
18. Rosenbloom ST, Geissbuhler AJ, Dupont WD, Giuse DA, Talbert DA, Tierney WM, et al. (2005). Effect of CPOE user interface design on user-initiated access to educational and patient information during clinical care. J Am Med Inform Assoc Association. 12(4):458-473.
19. Kovacs RJ, Lagarde M, Cairns J. (2020). Overconfident health workers provide lower quality healthcare. J Economic Psychol. 76:102213.
20. Berner ES, Graber ML. (2008). Overconfidence as a cause of diagnostic error in medicine. Am J Med. 121(5):S2-S23.
21. Cassam Q. (2017). Diagnostic error, overconfidence and self-knowledge. Palgrave Commun. 3:17025.
22. Dunning D. (2011). The Dunning–Kruger effect: On being ignorant of one's own ignorance. Adv Experiment Social Psychol. 44 : 247-296.
23. Kruger J, Dunning D. (1999). Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Personal Social Psychol. 77(6):1121.