Emilia Soroko, Maciej Borzyszkowski, Paweł Kleka, Klaudia Chmielak, Zuzanna Pawlak, Maja Skory Attitudes towards psychological online interventions (APOI) in relationship with personality and virtual social integration. Psychometric properties of the APOI questionnaire [Postawy wobec psychologicznych interwencji online (APOI) w powiązaniu z osobowością i wirtualną integracją społeczną. Właściwości psychometryczne kwestionariusza APOI]

Artykuł jest dostępny na warunkach międzynarodowej licencji 4.0 (CC BY-NC-ND 4.0).

Introduction

Online psychological interventions in healthcare

Mental health services in many countries are not sufficiently accessible to those in need due to financial cost, time and geographical (transport) constraints (Mohr et al., 2006). Attempts are being made to control burgeoning healthcare costs and to better organize services by integrating digital treatments into national mental health services (McDonald et al., 2020). The advantages of web-based interventions in the field of mental health are that they are low-threshold (reaching many people in need), flexible (adapting to their needs), and cost-efficient (both in them-selves and in terms of reducing the economic damage resulting from individuals not receiving the psychological help they need) (Hennemann, Beutel, Zwerenz, 2017). As an important advantage, the burden on existing healthcare systems connected with anonymity is also decreased (Barak et al., 2008), along with possible emotional costs such as difficulty in sharing personal stories (Mohr et al., 2006). Moreover, online interventions may be a good starting point for therapy in depression or implemented as a step in the therapeutic process—for example, where there is a waitlist (Cuijpers et al., 2011). Another benefit of psychological online interventions (POIs) is their applicability in the time of the COVID-19 pandemic, as the number of people experiencing psychological difficulties is increasing while help delivery has switched to remote mode (Békés, Aafjes-van Doorn, 2020). These advantages are a remedy for many of the barriers to getting psychological help to people in need: hence the drive to organize and implement online help. The potential of POIs is seen either as a complement to traditional services (blended treatment) or as a standalone form of intervention, serving as a viable alternative to existing treatments (Cipolletta, Mocellin, 2018). As Teles and colleagues (2020) wrote, successful implementation of online interventions seems to be determined by a complex range of factors, including technological, institutional, contextual and psychological. The authors attribute great importance to the latter and indicate attitudes and preferences towards online interventions.

There are many different terms for professional mental health services that are associated with the medium of the internet. General terms such as e-mental health, tele-mental health, digital treatments, internet-based interventions, psychological online intervention, for example, appear in the literature. The meanings of these terms are not the same, but in this paper we will use the term “psychological online intervention” (POI), understood broadly as encompassing a variety of modalities of interaction, including both online contact with a therapist and chatbot or gamified autonomous therapy (e.g. Barak, Klein, Proudfoot, 2009; McDonald et al., 2020). At the same time, we want to focus on interventions that are psychological in nature, which can have the status of more complex programmes or individual small-scale support interventions that support recovery and are delivered using the internet.

The value of POIs for the prevention and treatment of mental health problems is due to their evidence-based status (e.g. Hennemann, Beutel, Zwerenz, 2017). Meta-analyses have shown that online psychological services are comparable to face-to-face options in their effectiveness in treating mental illness. Barak and colleagues (2008) in their meta-analysis calculated a medium effect (.53) that was only slightly lower than the average face-to face therapy effect, and they expressed strong support for the adoption of POIs as a legitimate therapeutic activity. Specifically, the internet-based interventions were effective in reducing anxiety symptoms and increasing remission rates, but not effective in reducing depression symptom severity (Ye et al., 2014). Moreover, Andrews and colleagues (2018) concluded (after calculating the mean effect size as g = .80), that internet-delivered cognitive behavioural therapy for anxiety and depressive disorders is effective (in both the short and the long term), acceptable and practical health care.

Although internet-based interventions might be feasible treatment alternatives or complements, the vast majority of research has dealt only with cognitive-behavioural approaches (de Bitencourt Machado et al., 2016) aimed at symptom reduction. Moreover, there remains doubt as to whether POIs are useful and effective in severe cases of psychopathology.

Attitudes towards online psychological interventions

There is no doubt that many individuals appear to benefit from online therapy or other forms of POI (Hanley, 2021).This does not mean that everyone is willing to avail themselves of it. For this reason, it is worth addressing the issue of preferences for the form of help and attitudes towards POIs. Attitudes towards POIs are relatively enduring and general evaluation of the online interventions. Attitudes are rather summary evaluations (based on personal experiences) and are used to explain and predict health behaviours, including adherence to health-related interventions (Teles, Ferreira, Paúl, 2020). Preferences, on the other hand, relate more to situations of real or imagined choice between online and face-to-face options.

Research on attitudes towards POIs has been conducted among different groups: patients, mental health professionals and stakeholders (Hennemann, Beutel, Zwerenz, 2017; Topooco et al., 2017; Kuso et al., 2021). The attitudes expressed range from acceptance to resistance and barriers (Hennemann, Beutel, Zwerenz, 2017). For example, stakeholders showed greater acceptance of blended treatment compared to standalone internet treatments (Topooco et al., 2017). They also indicated the lack of personal contact, security, privacy and trust concerns, and highlighted needs such as target group appropriateness and the use of motivational tools (Kuso et al., 2021). Psychologists share concerns about technical issues, the confidentiality of client information, suitability and safety of users, and the therapeutic relationship (Cipolletta, Mocellin, 2018).

When patients are confronted with the choice of whether they prefer online or face-to-face therapy, despite the various conveniences of the remote form, they often choose or declare a preference for face-to-face therapy (e.g. Rochlen, Beretvas, Zack, 2004; Klein, Cook, 2010). Cipolletta and Mocellin (2018) collected the factors influencing attitudes towards POIs that are identified in research. They listed among them the theoretical orientation of the psychologist, prior knowledge (about the effectiveness and accessibility of online therapies), experience, and comfort with technology. In a study conducted by Apolinário-Hagen and colleagues (2018), participants assessed therapist-guided internet interventions as helpful and declared a willingness to uptake internet interventions. However, they did not treat online and face-to-face therapies as equivalent. The notion that attachment avoidance and stress are related to an e-preference (Apolinário-Hagen et al., 2018) could suggest the defensive role of such a preference. Moreover, almost 20 per cent of participants prefer not to use internet interventions and should be able to benefit from face-to-face assistance. It is also noted that, often contrary to clinicians’ expectations, patients perceive online intervention as a less inhibiting medium, enabling greater disclosure and interaction, and young patients are willing to engage with digital interfaces (McDonald et al., 2020), especially if they enjoy them on a daily basis (Sweeney et al., 2019). In a study of online interventions for eating disorders, variables associated with a preference for online therapy were: 1) not currently receiving psychotherapy; 2) more positive attitude; and 3) greater stigma associated with previous professional help-seeking (Linardon et al., 2020). There are therefore certain personal preferences and experiences specific to certain psychopathologies wherein POIs may be particularly worthwhile ways of helping.

Attitudes towards online psychological help and interventions are crucial for their effectiveness. First of all, positive attitudes were found to be related to higher efficacy of online interventions by optimizing the benefits in the process of its implementation (Schröder et al., 2018). Beliefs around the interventions may affect or moderate the potential psychological benefits as well as the engagement level (Schröder et al., 2015).

For online interventions, sustaining patient engagement in the therapy is an important issue. Important are 1) rates of uptake (entering the service) and possible non-use attrition (failing to access the intervention), and 2) dropout attrition (ceasing the service, discontinuing) (Casey, Clough, 2016). The dropout rate is estimated at 30 per cent on average, reaching up to 80 per cent (Melville, Casey, Kavanagh, 2010; McDonald et al., 2020; Teles, Ferreira, Paúl, 2020). Assessing attitudes prior to treatment might help identify suitable users (Schröder et al., 2018). Teles and colleagues (2020) identified a rationale for measuring attitudes towards online interventions: 1) screening and selection (in order to identify patients who would benefit the most); 2) better matching of participants’ attitudes and preferences with intervention delivery modes; 3) informing and educating potential users of online interventions about this delivery mode, especially about negative attitudes; and 4) improving online interventions based on the detection of negatively evaluated features and content. Making the measurement of attitudes a common procedure could contribute to the higher effectiveness of POIs and reduction in the various forms of attrition.

As it is likely that e-mental health interventions have the potential to become more than just temporary replacements of face-to-face consultations in times of pandemic crisis, it is of great importance to listen to the expectations and concerns of all parties involved, as well as to undertake research on internet-based health services and increasing public knowledge (e.g. Klein, Cook, 2010; Hollis et al., 2015; Apolinário-Hagen et al., 2018; De Witte et al., 2020). Developing knowledge of measurement and diagnostic methods is necessary to match forms of help to patients’ needs, to prevent different forms of attrition, and to gain knowledge and inform the community about help options.

Measurement of Attitudes towards Psychological Online Interventions

The Attitudes towards Psychological Online Interventions (APOI) questionnaire was originally developed by Schröder and colleagues (2015) in order to examine attitudes towards POIs among depressive patients. The authors of the original APOI questionnaire approached the attitudes as consisting of three main elements—cognitive, affective and behavioural components—whereas the acceptance of online interventions was defined as a cognitively-based positive attitude towards the interventions. During the construction of APOI items, the authors focused mainly on the cognitive aspect. The final version of the APOI questionnaire consists of 16 items, created on the basis of literature and existing tests, and based on five mechanisms contributing to the success of psychotherapy: therapeutic alliance, resource activation, problem actualization, motivational clarification, and mastery (Smith, Grawe, 2003). A statistical cross-validation of the factor structure within a confirmatory factor analysis (CFA) suggested that attitudes towards POIs can be represented in a hierarchical model including four first-order factors (with four items each) loading on one second-order global factor. The APOI consists of four subscales: scepticism and perception of risks, confidence in effectiveness, technologization threat, and anonymity benefits.

The anonymity benefits factor comprises items referring to users’ beliefs about online interventions in comparison to the traditional therapy model. The items concern difficulty disclosing personal problems to a therapist and the social stigma related to attending therapy. It may be seen as threatening to some due to the fear of mental illness social stigmatization (Vogel, Wade, Ascheman, 2009). A potential advantage of POIs is the possible lack of stigmatization risk during therapy with a therapist. Another difficulty in therapy could be the need to disclose personal problems to someone else. Research suggests that some of the attitudes towards face-to-face therapy refer to concerns about the therapeutic alliance (Moritz et al., 2013). The participants in the Moritz’s study feared that the therapist would make fun of them or would not like them. These concerns may be diminished when using POIs.

The technologization threat dimension refers to scepticism about the online form of the intervention and its effectiveness in contrast to the traditional face-toface therapy. The items link to aspects such as motivation for the treatment, understanding the psychological concepts, obtaining professional help in crisis situations, and acquiring new skills to manage everyday life better thanks to the therapy. Lack of personal contact with a therapist in POIs may be perceived as a disadvantage by some. The majority of the studies show that contact with a therapist is related to better outcomes from online interventions (Andersson, Titov, 2014). This is in line with a well-supported finding, that the therapeutic alliance is more important for the treatment outcomes than the techniques and theoretical background (Ahn, Wampold, 2001). Reduced engagement in POIs has been found to be linked with barriers such as technologization fatigue, beliefs about limited benefits from online interventions, or worries about committing too much time to sitting at a computer and neglecting everyday duties (Donkin, Glozier, 2012).

The confidence in effectiveness factor refers to the belief that POIs are effective interventions that could be inspiring and helpful in recognizing and approaching one’s struggles. Previous research suggests that treatment expectations and satisfaction with the intervention’s content are some of the factors that predict adherence to the online intervention (Beatty, Binnion, 2016). Therefore, it seems crucial to assess these aspects when studying opinions about POIs. There is research evidence suggesting that internal motivation and belief in the treatment format accounts for a significant part of the treatment’s outcomes and adherence (Alfonsson, Olsson, Hursti, 2016). A qualitative study conducted by Gerhards and colleagues (2011) suggested that one of the three key factors influencing adherence to online selfhelp cognitive-behavioural interventions for depression— beliefs about the Reed for people contact, the extent to which the interventions were relevant to the problem experienced, and motivation for participation in the online programme—was a social factor (Gerhards et al., 2011). Another study suggested that perceived personal benefit could be a motivator in persistence with online interventions and that a lack of personal benefit could be a factor leading to reduced persistence (Donkin, Glozier, 2012).

The scepticism and perception of risks dimension refers to a lack of confidence in the long-term effectiveness of POIs and concerns about receiving professional support from them. The items also relate to the fear of increased loneliness while using POIs and difficulty implementing changes in life from online interventions. As previous studies have suggested, intrinsic motivation for treatment or an intervention and belief in the treatment format affects its outcomes and adherence (Alfonsson, Olsson, Hursti, 2016). Lack of therapist guidance in online interventions could be difficult for some users, as research evidence suggests that the presence of the therapist and his or her contribution to the therapeutic alliance largely affect the psychological outcomes (Palmqvist, Carlbring, Andersson, 2007; Del Re et al., 2012). However, some studies show that the therapist’s support may be more practically related than therapeutically oriented (Andersson, Titov, 2014). Another factor behind the scepticism around POIs among users could be a lack of belief in receiving professional help. Indeed, the presence of a therapist could allow more suitable treatment or intervention for a particular patient, providing a diagnosis and directing the patient to other health services if needed (Andersson, Titov, 2014).

Aims of the current study

Attitudes towards online interventions remain poorly understood, and the assessment of attitude prior to starting an intervention is not yet a common procedure, in part because adequate measures are missing (Teles, Ferreira, Paúl, 2020). The main aim of this article is to make one of the tools more accessible research-wise and diagnostically, so that we can encourage patients to take up internet-based treatment, while understanding the arguments and fears before online help, as well as the expectations that it cannot fulfil. We achieve this goal by translating the APOI and determining the psychometric properties of the Polish version, focusing on internal and criterion validity. Additionally, we broaden the object of reference of attitudes towards psychological interventions. Previously, POIs used either standalone or as guided programmes that mostly feature cognitive behavioural methods have been studied, whereas in our study we asked about attitudes towards online interventions more broadly, encompassing web-based interventions, online counselling and therapy, internet-operated therapeutic software, and other online activities (e.g. Barak, Klein, Proudfoot, 2009). We met our objectives by conducting two separate studies.

Study 1

Aims

We examined the factor structure of the Polish version of APOI (APOI-PL). We expected that a four-factor solution would provide the best fit. We also investigated the reliability and the discriminatory power of items. Because research shows that the way women and men use and appraise psychological help may differ (e.g. Nam et al., 2010; Liddon, Kingerlee, Barry, 2018), we analysed invariance by gender.

Method

Participants and procedure

Three hundred and four participants were recruited from Polish citizens through dissemination of links (Google Forms) to the research via the internet and temporary sponsored advertisements on Facebook. Data collection started at the beginning of the pandemic in Poland (March 2020) and responses were received for ten months. The surveys were aimed at adult volunteers and participation was unremunerated. Participants completed an online questionnaire, providing information about their experiences with POIs and demographic data. As the time of the pandemic was a dynamic period, at some point in the survey we included a question about whether the pandemic affected attitudes towards psychological support. The data from a subsample of 115 participants showed that 77.39 per cent declared that the pandemic had not influenced their attitude or had influenced it very little, while the remaining 22.61 per cent explicitly noted or admitted the possibility of such an influence on attitude towards online psychological interventions. The sample characteristics are presented in Table 1.

Table 1. Participant demographics (Study 1: N = 304)

Barak and colleagues (2009) distinguished four types of online psychological help: web-based interventions (primarily self-guided intervention programs that are executed by means of a prescriptive online program operated through a website); online counselling and therapy (individual or group contact, using either a synchronous or an asynchronous communication mode); internet-operated therapeutic software (advanced computer capabilities such as artificial intelligence principles, e.g. with robotic simulation, rule-based expert systems, gaming, 3D virtual environments); and other online activities (personal blogs, participation in support groups via chat, audio, or webcam communication channels, the use of online assessments, and accessing health-related information via information sites and podcasts). Among the forms of psychological help in the study group, the most familiar were online counselling and therapy and other online activities (e.g. blogs, podcasts). Less familiar were internet-operated therapeutic software and web-based interventions. It is also worth mentioning that among those using psychotherapeutic help, 61 people used pharmacotherapy; among the remaining people, 28 use pharmacotherapy.

Materials

Translation. The APOI questionnaire is designed for people both with and without experience of online psychological help. Both versions of the questionnaire were translated into Polish by one of the authors (KCH) directly from the source. Together with the authors of the original version, we did a series of back-translations before arriving at an agreed draft version. For socio-cultural and linguistic validation, we consulted independent experts on the translated items, who checked which areas the questionnaire items referred to and helped to refine the content of the questions. In Study 1, we used the version for participants that had no experience of POI themselves but wanted to share their opinion on it.

Attitudes towards Psychological Online Interventions Questionnaire (APOI). In the research we used APOI-PL, consisting of 16 items. Items are divided on four subscales: scepticism and perception of risks (SCE); confidence in effectiveness (CON); technologization threat (TET); and anonymity benefits (ABE) (Schröder et al., 2015). Using a five-point scale ranging from 1 (strongly disagree) to 5 (strongly agree), participants indicated their attitude towards POIs.

Analyses

Based on the Cook’s distance index values, 16 outliers (Δ > 1) were removed, resulting in a final sample of 288 observations. In order to assess the reliability of the APOI-PL scale, the McDonald’s omega coefficient was determined, and, for the subscales, Cronbach’s alpha, taking a value above .70 as a satisfactory measure of reliability and internal consistency. Confirmatory factor analysis was used with maximum likelihood (ML) estimation to analyse construct validity; as no multivariate normality of the data distribution was obtained (Mardia skewness = 1499.9, p < .001, Mardia kurtosis = 15.2, p < .001), the robust Satorra-Bentler chi-square test was used. It was assumed that comparative fit index (CFI) and Tocker-Lewis index (TLI) indices with values above .90 and RMSEA below .08 would indicate a good fit of the data to the model and thus confirm the theoretical accuracy of the APOI construct (Hu, Bentler, 1996; Whittaker, 2016). In addition, an analysis of invariance between men and women was conducted. In a first step, configural invariance (factor structure consistency) was tested, followed by metric invariance, factor loadings consistency, and finally scalar invariance (fixed intercept) and strict invariance (fixed residual). It was assumed that a decrease in CFI below .01 and an increase RMSEA above .015 indicates a lack of measurement equivalence (Chen, 2007).

Results

Confirmatory factor analysis

The four-factor model derived from Schröder and colleagues (2015) was tested. The model of the Polish version of APOI achieved a good fit to the empirical data (Figure 1, Table 2). As shown in Figure 1, the subscales CON and ABE (that represent positive attitudes towards online interventions) correlate negatively with TET and SCE. The strongest factor loadings are observed in the CON (range: .72 to .90) and the weakest in the ABE (range: .37 to .71) subscales.

Figure 1. Model CFA for APOI scale

Notes: SCE – Scepticism and perception of risk; CON – Confidence in effectiveness; TET – Technologization threat; ABE – Anonymity benefits. Robust values of fit indices: χ2 (98) = 199.659, p < .001, CFI = .927, TLI = .910, RMSEA = .060, 90% CI = [.049, .071].

Invariance by gender

An invariance test conducted on a sample of men and women (N = 283) provided information on the gender loading of APOI questionnaire items (Table 2).

Table 2. Invariance for APOI scale by gender

Detailed analysis of differences in CFI and RMSEA between adjacent models showed differences between model 1 and model 2, implying a lack of metric equivalence between measures in the female and male groups. The difference concerned two items from the ABE subscale in the comparison of online interventions with traditional therapy. This can be understood as a cross-gender difference in disclosing personal problems to a therapist and the social stigma related to attending therapy. Women have higher loadings in the statement showing the advantages of online interventions when revealing feelings (item 3: “By using a POI, patients can reveal their feelings more easily than with a therapist”) and telling friends (item 2: “Patients would be more likely to tell their friends that they use a POI than that they visit a therapist”). Further analysis of the differences between the metric partial model and the scalar model testing for equality of intercepts indicated that there was no difference in mean scores between men and women. Also there was no difference in items’ residuals based on comparison of the scalar and the strict models. It is worth noting that the differences in CFI and RMSEA values between models do not exceed the accepted thresholds, indicating that the observed gender bias of the APOI scale is relatively small.

Reliability of APOI-PL subscales

Reliability in the sample, as measured by the McDonald’s omega coefficient, was ω = .90, and the level of common variance reached EVC = 44%. Measures of consistency were also determined for individual scales based on Cronbach’s alpha (Table 3).

Table 3. Reliability of APOI and reliability of subscales after item deletion

The CON subscale shows high internal consistency, while the TET subscale shows acceptable consistency. Given the shortness of the subscales, acceptable levels are shown by SCE and ABE, though below the value of .70. The results are comparable to those obtained by the authors of the original tool.

Item discrimination power indices

Across all items, the lowest discriminatory power characterizes APOI items 4 and 15 on the SCE scale and APOI 2 on the ABE scale. These items also adequately have lower factor loadings in both the female and male groups (Table 4).

Table 4. Discrimination power of items and CFA loading for them

Intercorrelations

Intercorrelations among the subscales were calculated with Spearman’s rho (Table 5). Moderate and low correlations were reasonable and congruent with pre[1]vious findings. The highest interrelations were observed between SCE and CON  (–.564), and SCE and TET (.519). SCE and ABE were not related. The interrelations  observed in our study are very similar to the original study (Schröder et al., 2015);  however, we obtained slightly higher (but still low) correlation between CON and  ABE, and TET and ABE. We also found lower correlation between APOI POS and  ABE than in a previous study (Schröder et al., 2015).

Table 5. Intercorrelations of the APOI subscales (N = 304)

Study 2

Aims and hypotheses

In this study, our aim was to determine the criterion validity of the APOI-PL scale. As the tool is novel, we chose variables for which there is theoretical and empirical justification that they may be related to attitudes towards POIs. These were the Big Five personality traits and virtual social integration (feelings of connectedness to others in sustaining online relationships). We expected that positive attitudes towards psychological interventions in general and confidence in effectiveness (CON) in particular would be related to openness. In contrast, subscales expressing reserve towards online help (i.e. TET and SCE) would be negatively correlated with openness. We relied on the fact that openness to experience was significantly and positively related to perceived ease of use of the technology (Svendsen et al., 2013) and internet usage, but only among Baby Boomers, born 1946–1964 (Roos, Kazemi, 2021). Also, we hypothesized a positive relationship between a positive attitude towards online psychological help and extraversion, given results indicating extraversion is positively associated with using the internet for information, duty, leisure and social activities (e.g. Roos, Kazemi, 2021). However, extraversion decreased the likelihood of preferring an internet-based mental health service, and neuroticism, extraversion, and conscientiousness from the NEO-Five Factor Inventory (NEO-FFI) might also be connected with attitudes towards POI (March et al., 2018). Similarly, Klein and Cook (2010) found that “non e-preferers” had significantly higher scores on extraversion than “e-preferers”. Where virtual social integration (VSI) is concerned, we hypothesized that valuing social experiences online would correlate positively with the confidence in effectiveness scale (CON) and negatively with scepticism and perception of risk (SCE). This is because people believe in the effectiveness of therapy, because having a sense of connectedness online means that it is likely that they are already receiving support and know that it works for them (Hill, Weinert, 2004; Ward, Tracey, 2004).

Method

Participants and procedure

Participants completed a battery of online questionnaires and additional questions about the usage of internet interventions with the use of Google Forms in response to courtesy notices on Facebook profiles. The study started at the onset of the pandemic in Poland and was carried out over three months. Participation in the study was voluntary and anonymous. The advertisement was mainly answered by female students from big cities. Detailed sample characteristics are presented in Table 6.

Table 6. Participants demographics (Study 2: N = 164)

Materials

Attitudes towards Psychological Online Interventions Questionnaire (APOI-PL). This is described in Study 1. Two versions of the questionnaire were used. Individuals answered the version that matched them based on a qualifying question about whether they had personal experience of POIs and 28.8% had such an own experience.

International Personality Item Pool NEO-Five Factor Inventory-50. The International Personality Item Pool NEO-Five Factor Inventory-50 (IPIP-Neo FFI) is a Polish adaptation of a self-report measure assessing the Big Five personality traits based on a lexical approach (Strus, Cieciuch, Rowiński, 2014). Each scale (extraversion, agreeableness, conscientiousness, emotional stability, openness to experience) consists of ten items, giving a total of 50 items. Participants answer using a five-point scale ranging from 1 (describes me completely inaccurately) to 5 (describes me completely accurately). The internal consistency estimate (Cronbach’s α) for the present study sample is as follows: emotional instability – .887; extraversion – .879; openness to experience – .751; agreeableness – .780; conscientiousness – .862.

Virtual Social Integration Questionnaire. The Virtual Social Integration Questionnaire (VSI) is a measure created by Klej and Jurek (2017) to assess a sense of belonging to an online community (Klej, Jurek, 2017). The questionnaire reveals a declared sense of connectedness to others in online relationships, quantified in terms of perceived benefits and feelings of pleasure from online relationships with others. Using a five-point scale ranging from 1 (strongly disagree) to 5 (strongly agree), participants indicate their sense of belonging to an online community across ten items. It is expected that participants with a stronger sense of belonging to an online community will have a more positive attitude towards POI. The internal consistency estimate for the present study sample is Cronbach’s α = .869.

Analyses and results

The Pearson’s r correlation coefficient was used to test the hypotheses about the expected relationships between attitudes towards POI and personality and social network embeddedness. The results are only partly in line with the hypotheses (Table 7). Overall positive attitudes towards POIs are not associated with openness, but the CON (confidence in effectiveness) subscale showed a low positive significant correlation with openness and virtual social inclusion (VIS). As expected, subscales expressing reserve (TET and SCE) were found to be negatively correlated with openness. We observed no relationship between positive attitudes towards POI and extraversion. We expected the severity of virtual social interactions (VSI) to correlate positively with the CON scale (we obtained a low correlation) and negatively with SCE (there was no correlation in our study). The only personality variable that showed a low positive correlation with a positive attitude was agreeableness, which is a result worth looking at in further research. More specifically, agreeableness is negatively correlated with scepticism and perception of risk (SCE) and technologization threat (TET), and positively correlated with confidence in effectiveness (CON).

Table 7. Correlations between APOI-PL, IPIP and VSI (N = 162)

Discussion

First of all, the theoretical accuracy of the Polish version of the APOI was examined by applying confirmatory factor analysis. The model of the Polish version of the scale fitted the empirical data well both for the whole group of respondents and separately for the groups of men and women. The model fit was sufficient to accept factorial accuracy of the APOI (CFI > .90, RMSEA < .08), which confirms the validity of separating the four theoretically justified subscales of the APOI scale. Comparing the obtained results with the results of other authors, it can be stated that the model of the Polish version of the scale estimated on the whole group of subjects fitted the data slightly less well than the model presented by the authors of the original APOI scale (Schröder et al., 2015). An analogous pattern of correlations between factors representing individual subscales was also noted, which is consistent with previous research findings. Reliability coefficients for the individual subscales similar to the scale authors were also obtained, with CON and TET having higher internal consistency.

Measurement invariance by gender (female, male) was also demonstrated. Four basic types of measurement invariance were demonstrated, i.e. configural, metric, scalar and strict, taking into account the difference between men and women for items 2 and 3 of the ABE subscale. Women are more convinced than men that patients can reveal their feelings more easily with POI than with a therapist. Similarly, they are more convinced than men that patients are more likely to tell their friends that they use a POI than that they visit a therapist. These two statements in men are less strongly associated with anonymity benefits (ABE). This may mean that men find it more difficult to consider a POI sufficiently anonymous. This result fits with the recognition that men find it harder to engage in help-seeking behaviour (e.g. Liddon, Kingerlee, Barry, 2018), but also shows that, in men, the belief that they are sufficiently safe when seeking help may be harder to obtain even in online therapy. Researchers have noted that men must first recognize depression and then overcome considerable perceived and internalized stigma to ask for help (House et al., 2018) or they need a “man friendly” therapeutic orientation (Cole et al., 2019). The importance of trust and confidentiality seems to be a very important factor when organizing support for men (Sagar-Ouriaghli et al., 2020). In conclusion, given the results of the analysis of invariance, it can be assumed that women’s and men’s scores on the APOI will be the same across three subscales, with women scoring higher on the ABE. It is worth monitoring whether this invariance continues in the course of further research.

The criterion validity analysis yielded results only partially in line with the hypotheses. Although we expected that positive attitudes towards POIs would be related to openness, the only low significant correlation was observed with the IPP-Neo FFI agreeableness. This result, not previously assumed, directs our attention to two problems. Firstly, agreeableness may be expressed in accepting various cues or experiences suggested by others, lowering scepticism towards those experiences. The results also show that with higher agreeableness, the sense of technologization threat may decrease. This may indicate a kind of personality vulnerability that should suggest caution and sensitize helpers to seemingly voluntary consent in people who have high agreeableness but POI is not an intervention well suited to them. Secondly, however, IPIP agreeableness, as a personality trait in the context of POIs, may foster positive outcomes in psychotherapy, especially through good therapeutic alliances (e.g. Bucher, Suzuki, Samuel, 2019) and satisfaction with the support (Pornsakulvanich, 2017).

The correlation between the confidence in effectiveness (CON) and IPIP openness was confirmed, but was low. As expected, IPIP openness was negatively related to technologization threat (TET) and scepticism and perception of risk (SCE). It is likely that openness to experience allows one to use new technologies with greater ease, as has been reported in other studies (Svendsen et al., 2013; Roos, Kazemi, 2021). In the literature, reports of associations between extraversion and attitude towards online psychological help are contradictory. Roos and Kazemi (2021) found positive correlations with using the internet for information, duty, leisure and social activities. However, there is also evidence that extraversion decreases the likelihood of preferring an internet-based mental health service (Klein, Cook, 2010; March et al., 2018). Our research shows that there is no support for linking extraversion to attitude towards POIs. We also noted that IPIP emotional instability was weakly negatively related with anonymity benefits (ABE), showing that anxious doubts may contribute to the belief that confidentiality is difficult with POIs.

The hypothesis that VSI would correlate positively with the confidence in effectiveness scale (CON) was confirmed; however, the correlation is low. We did not find any negative correlation with scepticism and perception of risk (SCE). It is therefore likely that positive online social experiences slightly increase efficacy beliefs, which is in line with reports in previous research (Hill, Weinert, 2004; Ward, Tracey, 2004).

Limitations and future directions

The research presented here has several limitations that affect the interpretation of the results. In both our studies, the sample included volunteers whose participation in the online survey was not monitored, which may have contributed to less accurate data. At the same time, effort was made to prepare the data for analysis by checking for random responses and removing outliers. Compared to the original study, the age of the respondents in our study was lower, and women predominated. Moreover, we broadened the object of attitudes in the study, distinguishing four groups: web-based interventions, online counselling and therapy, internet-operated therapeutic software, and other online activities (Barak, Klein, Proudfoot, 2009). While there are advantages to this decision, it is worth bearing in mind that these forms are different enough to elicit different attitudes in people. Besides, nowadays the forms of assistance have further diversified, giving new criteria for classification (see McDonald et al., 2020). In our study, attitudes primarily related to online counselling and therapy.

An important context for the results is the impact of the COVID-19 pandemic on outcomes. The data collection period for our study coincided with the first two waves of the COVID-19 pandemic in Poland (March–December 2020). A larger number of people began to have experience of online psychological help and thus had the opportunity to form an opinion about it and confront their existing views with their actual online session. However, this confrontation may also have had a negative effect due to the compulsion to change the form of therapy, as a result of which an aversion to online psychological help may have arisen. Besides, the actual impact of the pandemic on attitude scores is not known.

In addition to this, in Study 1 we used a version of the tool aimed only at people who express an attitude but have not previously participated in online surveys. Further research should target not only the general population, but also potential patients and people who have experience with POIs. The correlations obtained in Study 2 should contribute to further exploration of criterion relevance. It is worth considering the modifying role of categories of internet usage and generation cohorts when examining personality (e.g. Roos, Kazemi, 2021). A limitation of Study 2 is also the choice of validation tools. The field of diagnosing attitudes towards therapy, psychoeducation and online interventions is constantly expanding, to include, for example, the Attitudes toward Internet-Based Psychotherapy scale (ETAM; Apolinário-Hagen et al., 2017) and the Online Psychoeducational Interventions – Brief Attitudes Scale (OPI-BAS; Teles, Ferreira, Paúl, 2020). Further research will need to check the criterion relevance in relation to the above-mentioned tools.

Conclusion

The Polish version of the APOI scale has adequate psychometric properties indicating its high accuracy and reliability. A satisfactory fit of the postulated model to empirical data was demonstrated in confirmatory factor analysis; measurement invariance between women and men was also noted, and high internal consistency indices were obtained. Women were characterized by a stronger association with certain aspects of the benefits of anonymity (items 2 and 3; indices) and ABE (latent trait). The gender invariant nature of the measure has not been demonstrated previously, and our results here suggest the need to monitor it in future studies.

Positive attitudes toward POIs were weakly positively associated with IPIP agreeableness. Anonymity benefits (ABE) were weakly negatively related to IPIP emotional instability and IPIP openness. Confidence in effectiveness (CON) was weakly positively related with IPIP openness, IPIP agreeableness and VSI. Scepticism and perception of risk (SCE) was negatively related to IPIP agreeableness and IPIP openness. Technologization threat (TET) was negatively related to IPIP agreeableness. This configuration of correlations is reasonable, although the relationships obtained are weak. Certainly, the analysis of criterion validity requires further research.

References

Ahn, H.N., & Wampold, B.E. (2001). Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy. Journal of Counseling Psychology, 48(3), 251, doi: 10.1037/0022-0167.48.3.251

Alfonsson, S., Olsson, E., & Hursti, T. (2016). Motivation and treatment credibility predicts dropout, treatment adherence, and clinical outcomes in an internet- based cognitive behavioral relaxation program: A randomized controlled trial. Journal of Medical Internet Research, 18(3), e52, doi: 10.2196/jmir.5352

Andersson, G., & Titov, N. (2014). Advantages and limitations of Internet‐based interventions for common mental disorders. World Psychiatry, 13(1), 4–11, doi: 10.1002/wps.20083

Andrews, G., Basu, A., Cuijpers, P., Craske, M.G., McEvoy, P., English, C.L., & Newby, J.M. (2018). Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: An updated meta-analysis. Journal of Anxiety Disorders, 55, 70–78, doi: 10.1016/j.janxdis.2018.01.001

Apolinário-Hagen, J., Harrer, M., Kählke, F., Fritsche, L., Salewski, C., & Ebert, D.D. (2018). Public attitudes toward guided internet-based therapies: Web-based survey study. JMIR Mental Health, 5(2), e10735, doi: 10.2196/10735

Barak, A., Hen, L., Boniel-Nissim, M., & Shapira, N.A. (2008). A comprehensive review and a meta-analysis of the effectiveness of internet-based psychotherapeutic interventions. Journal of Technology in Human Services, 26(2–4), 109–160, doi: 10.1080/15228830802094429

Barak, A., Klein, B., & Proudfoot, J.G. (2009). Defining internet-supported therapeutic interventions. Annals of Behavioral Medicine, 38(1), 4–17, doi: 10.1007/s12160- 009-9130-7

Beatty, L., & Binnion, C. (2016). A systematic review of predictors of, and reasons for, adherence to online psychological interventions. International Journal of Behavioral Medicine, 23(6), 776–794, doi: 10.1007/s12529-016-9556-9

Békés, V., & Aafjes-van Doorn, K. (2020). Psychotherapists’ attitudes toward online therapy during the COVID-19 pandemic. Journal of Psychotherapy Integration, 30(2), 238–247, doi: 10.1037/int0000214

Bucher, M.A., Suzuki, T., & Samuel, D.B. (2019). A meta-analytic review of personality traits and their associations with mental health treatment outcomes. Clinical Psychology Review, 70, 51–63, doi: 10.1016/j.cpr.2019.04.002

Casey, L.M., & Clough, B.A. (2016). 8. Making and Keeping the Connection: Improving Consumer Attitudes and Engagement in E-Mental Health Interventions. In The Psychology of Social Networking Vol. 1 (pp. 90–103). De Gruyter Open Poland, doi: 10.1515/9783110473780-010

Chen, F.F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504, doi: 10.1080/10705510701301834

Cipolletta, S., & Mocellin, D. (2018). Online counseling: An exploratory survey of Italian psychologists’ attitudes towards new ways of interaction. Psychotherapy Research, 28(6), 909–924, doi: 10.1080/10503307.2016.1259533

Cole, B.P., Petronzi, G.J., Singley, D.B., & Baglieri, M. (2019). Predictors of men’s psychotherapy preferences. Counselling and Psychotherapy Research, 19(1), 45–56, doi: 10.1002/capr.12201

Cuijpers, P., Donker, T., Johansson, R., Mohr, D.C., van Straten, A., & Andersson, G. (2011). Self-guided psychological treatment for depressive symptoms: A meta- analysis. PLoS ONE, 6(6), e21274, doi: 10.1371/journal.pone.0021274

de Bitencourt Machado, D., Braga Laskoski, P., Trelles Severo, C., Margareth Bassols, A., Sfoggia, A., Kowacs, C., …, & Pigatto Teche, S. (2016). A psychodynamic perspective on a systematic review of online psychotherapy for adults. British Journal of Psychotherapy, 32(1), 79–108, doi: 10.1111/bjp.12204

De Witte, N.A., Carlbring, P., Etzelmueller, A., Nordgreen, T., Karekla, M., Haddouk, L., …, & Brugnera, A. (2020). Online consultations in mental healthcare during the Covid-19 outbreak: An international survey study on uptake and experiences. JMIR Preprints, 26541, doi: 10.2196/preprints.26541

Del Re, A.C., Flückiger, C., Horvath, A.O., Symonds, D., & Wampold, B.E. (2012). Therapist effects in the therapeutic alliance-outcome relationship: A restricted-maximum likelihood meta-analysis. Clinical Psychology Review, 32(7), 642–649, doi: 10.1016/j.cpr.2012.07.002

Donkin, L., & Glozier, N. (2012). Motivators and motivations to persist with online psychological interventions: a qualitative study of treatment completers. Journal of Medical Internet Research, 14(3), e91, doi: 10.2196/jmir.2100

Gerhards, S.A.H., Abma, T.A., Arntz, A., De Graaf, L.E., Evers, S.M.A.A., Huibers, M.J.H., & Widdershoven, G.A.M. (2011). Improving adherence and effectiveness of computerised cognitive behavioural therapy without support for depression: A qualitative study on patient experiences. Journal of Affective Disorders, 129(1–3), 117–125, doi: 10.1016/j.jad.2010.09.012

Hanley, T. (2021). Researching online counselling and psychotherapy: The past, the present and the future. Counselling and Psychotherapy Research, 21(3), 493–497, doi: 10.1002/capr.1238

Hennemann, S., Beutel, M.E., & Zwerenz, R. (2017). Ready for eHealth? Health professionals’ acceptance and adoption of eHealth interventions in inpatient routine care. Journal of Health Communication, 22(3), 274–284, doi: 10.1080/10810730. 2017.1284286

Hill, W.G., & Weinert, C. (2004). An evaluation of an online intervention to provide social support and health education. CIN: Computers, Informatics, Nursing, 22(5), 282–288, doi: 10.1097/00024665-200409000-00008

Hollis, C., Morriss, R., Martin, J., Amani, S., Cotton, R., Denis, M., & Lewis, S. (2015). Technological innovations in mental healthcare: Harnessing the digital revolution. The British Journal of Psychiatry, 206(4), 263–265, doi: 10.1192/bjp.bp. 113.142612

House, J., Marasli, P., Lister, M., & Brown, J.S. (2018). Male views on help‐seeking for depression: AQ methodology study. Psychology and Psychotherapy: Theory, Research and Practice, 91(1), 117–140, doi: 10.1111/papt.12144

Hu, L.T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55, doi: 10.1080/10705519909540118

Klein, B., & Cook, S. (2010). Preferences for e-mental health services amongst an online Australian sample. E-Journal of Applied Psychology, 6(1), doi: 10.7790/ejap. v6i1.184

Klej, A., & Jurek, P. (2017). Kwestionariusz Wirtualnej Integracji Społecznej (WIS) – konstrukcja i właściwości psychometryczne narzędzia mierzącego poczucie zintegrowania z innymi w przestrzeni Internetu. Polskie Forum Psychologiczne, 22(1), 139–158, doi: 10.14656/PFP20170108

Kuso, S., Nitsch, M., Zeiler, M., Simek, M., Adamcik, T., Dey, M., …, & Ebert, D.D. (2021). Stakeholders’ views on online interventions to prevent common mental health disorders in adults implemented into existing healthcare systems in Europe. European Journal of Public Health, 31(Supplement_1), i55–i63, doi: 10.1093/ eurpub/ckab043

Liddon, L., Kingerlee, R., & Barry, J.A. (2018). Gender differences in preferences for psychological treatment, coping strategies, and triggers to help‐seeking. British Journal of Clinical Psychology, 57(1), 42–58, doi: 10.1111/bjc.12147

Linardon, J., Shatte, A., Tepper, H., & Fuller‐Tyszkiewicz, M. (2020). A survey study of attitudes toward, and preferences for, e‐therapy interventions for eating disorder psychopathology. International Journal of Eating Disorders, 53(6), 907–916, doi: 10.1002/eat.23268

March, S., Day, J., Ritchie, G., Rowe, A., Gough, J., Hall, T., Yuen, C.Y.J., Donovan, C.L., & Ireland, M. (2018). Attitudes toward e-mental health services in a community sample of adults: Online survey. Journal of Medical Internet Research, 20(2), e9109, doi: 10.2196/jmir.9109

McDonald, A., Eccles, J.A., Fallahkhair, S., & Critchley, H.D. (2020). Online psychotherapy: Trailblazing digital healthcare. BJPsych Bulletin, 44(2), 60–66, doi: 10.1192/bjb.2019.66

Melville, K.M., Casey, L.M., & Kavanagh, D.J. (2010). Dropout from Internet‐based treatment for psychological disorders. British Journal of Clinical Psychology, 49(4), 455–471, doi: 10.1348/014466509X472138

Mohr, D.C., Hart, S.L., Howard, I., Julian, L., Vella, L., Catledge, C., & Feldman, M.D. (2006). Barriers to psychotherapy among depressed and nondepressed primary care patients. Annals of Behavioral Medicine, 32(3), 254–258, doi: 10.1207/ s15324796abm3203_12

Moritz, S., Schröder, J., Meyer, B., & Hauschildt, M. (2013). The more it is needed, the less it is wanted: Attitudes toward face‐to‐face intervention among depressed patients undergoing online treatment. Depression and Anxiety, 30(2), 157–167, doi: 10.1002/da.21988

Nam, S.K., Chu, H.J., Lee, M.K., Lee, J.H., Kim, N., & Lee, S.M. (2010). A meta-analysis of gender differences in attitudes toward seeking professional psychological help. Journal of American College Health, 59(2), 110–116, doi: 10.1080/ 07448481.2010.483714

Palmqvist, B., Carlbring, P., & Andersson, G. (2007). Internet-delivered treatments with or without therapist input: Does the therapist factor have implications for efficacy and cost? Expert Review of Pharmacoeconomics & Outcomes Research, 7(3), 291–297, doi: 10.1586/14737167.7.3.291

Pornsakulvanich, V. (2017). Personality, attitudes, social influences, and social networking site usage predicting online social support. Computers in Human Behavior, 76, 255–262, doi: 10.1016/j.chb.2017.07.021

Rochlen, A.B., Beretvas, S.N., & Zack, J.S. (2004). The online and face-to-face counseling attitudes scales: A validation study. Measurement and Evaluation in Counseling and Development, 37(2), 95–111, doi: 10.1080/07481756.2004.11909753

Roos, J.M., & Kazemi, A. (2021). Personality traits and Internet usage across generation cohorts: Insights from a nationally representative study. Current Psychology, 40(3), 1287–1297, doi: 10.1007/s12144-018-0033-2

Sagar-Ouriaghli, I., Godfrey, E., Graham, S., & Brown, J.S. (2020). Improving mental health help-seeking behaviours for male students: A framework for developing a complex intervention. International Journal of Environmental Research and Public Health, 17(14), 4965, doi: 10.3390/ijerph17144965

Schröder, J., Berger, T., Meyer, B., Lutz, W., Späth, C., Michel, P., …, & Moritz, S. (2018). Impact and change of attitudes toward Internet interventions within a randomized controlled trial on individuals with depression symptoms. Depression and Anxiety, 35(5), 421–430, doi: 10.1002/da.22727

Schröder, J., Sautier, L., Kriston, L., Berger, T., Meyer, B., Späth, C., …, & Moritz, S. (2015). Development of a questionnaire measuring Attitudes towards Psychological Online Interventions – the APOI. Journal of Affective Disorders, 187, 136–141, doi: 10.1016/j.jad.2015.08.044

Smith, E.C., & Grawe, K. (2003). What makes psychotherapy sessions productive? A new approach to bridging the gap between process research and practice. Clinical Psychology & Psychotherapy: An International Journal of Theory & Practice, 10(5), 275–285, doi: 10.1002/cpp.377

Strus, W., Cieciuch, J., & Rowiński, T. (2014). Polska adaptacja kwestionariusza IPIP-BFM- 50 do pomiaru pięciu cech osobowości w ujęciu leksykalnym. Roczniki Psychologiczne, 17(2), 327–346, https://ojs.tnkul.pl/index.php/rpsych/article/view/511

Svendsen, G.B., Johnsen, J.A.K., Almås-Sørensen, L., & Vittersø, J. (2013). Personality and technology acceptance: The influence of personality factors on the core constructs of the Technology Acceptance Model. Behaviour & Information Technology, 32(4), 323–334, doi: 10.1080/0144929X.2011.553740

Sweeney, G.M., Donovan, C.L., March, S., & Forbes, Y. (2019). Logging into therapy: Adolescent perceptions of online therapies for mental health problems. Internet Interventions, 15, 93–99, doi: 10.1016/j.invent.2016.12.001

Teles, S., Ferreira, A., & Paúl, C. (2020). Assessing attitudes towards online psychoeducational interventions: Psychometric properties of a Brief Attitudes Scale. Health & Social Care in the Community, doi: 10.1111/hsc.13227

Topooco, N., Riper, H., Araya, R., Berking, M., Brunn, M., Chevreul, K., Cieslak, R., Ebert, D.D., Etchmendy, E., Herrero, R., & Kleiboer, A. (2017). Attitudes towards digital treatment for depression: A European stakeholder survey. Internet Interventions, 8, 1–9, doi: 10.1016/j.invent.2017.01.001

Vogel, D.L., Wade, N.G., & Ascheman, P.L. (2009). Measuring perceptions of stigmatization by others for seeking psychological help: Reliability and validity of a new stigma scale with college students. Journal of Counseling Psychology, 56(2), 301–308, doi: 10.1037/a0014903

Ward, C.C., & Tracey, T.J. (2004). Relation of shyness with aspects of online relationship involvement. Journal of Social and Personal Relationships, 21(5), 611–623, doi: 10.1177/0265407504045890

Whittaker, T.A. (2016). Structural equation modelling. In K.A. Pituch, & J.P. Stevens (Eds.), Applied Multivariate Statistics for the Social Sciences: Analyses with SAS and IBM’s SPSS (pp. 639–733). New York, NY, USA.

Ye, X., Bapuji, S.B., Winters, S.E., Struthers, A., Raynard, M., Metge, C., …, & Sutherland, K. (2014). Effectiveness of internet-based interventions for children, youth, and young adults with anxiety and/or depression: A systematic review and meta-analysis. BMC Health Services Research, 14(1), 1–9, doi: 10.1186/ 1472-6963-14-313