Digital Divide : A Critical Context for Digitally Based Assessments

Student learning is increasingly taking place in digital environments both within and outside schooling contexts. Educational assessments are following suit, both to take Education Policy Analysis Archives Vol. 26 No. 51 SPECIAL ISSUE 2 advantage of the conveniences and opportunities that digital environments provide as well as to reflect the mediums of learning increasingly taking place in societies around the world. A social context relevant to learning and assessment in the digital age is the great differences in access to and competence in technology among students from different segments of societies. Therefore, access and competency in relation to technology become critical contexts for evaluations that rely on digitally based assessments. This chapter examines the digital divide between students from different segments of the society and discusses strategies for minimizing effects of digital divide on assessments of student learning. The research focuses on two types of demographic groups—gender and socioeconomic status (SES) groups—that have been highlighted in research on the digital divide. The research utilizes data from IEA’s International Computer and Information Literacy Study (ICILS) 2013 for Grade 8 students administered in 21 jurisdictions around the world. It thus provides an international perspective on digital divide as an important context for international assessments as well as assessments within jurisdictions such as Mexico that are conducting assessments in digitally based environments.


Digital Divide: A Critical Context for Digitally Based Assessments
In the 21 st century, abilities and skills associated with information, computing and technology (ICT) are central to school learning and assessment, success in the workplace-and, more broadly, effective functioning in society.In education, assessments of educational outcomes are increasingly being conducted in digitally based environments.In Canada and the USA, almost all educational assessments conducted at state/province or national levels have components that are administered on digital platforms.International assessments of educational outcomes such as the Programme for International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS) have already started or are gearing up for assessments administered in digital environments.In Mexico, both PISA and teacher evaluations are currently administered digitally.In these assessments, access to and use of technology in everyday life, work and school contexts constitute a critical context for assessment.In particular, information and computing technology (ICT) skills become an essential component of any digitally based assessment.In addition to inequities in such competencies having important implications for access to education and employment, they also have implications for how we interpret the results of educational assessments.The validity of inferences from educational assessments that are used for evaluating educational outcomes and the effectiveness of education systems is critically dependent on the degree to which performance on the assessments is an accurate indicator of targeted knowledge, skills and competencies rather than student familiarity with and skills in using technology.Therefore, documentation and understanding of ICT skills for different student groups are essential for educators around the world as they address inequities and interpret assessment results in digital environments.
The purpose of this research is to examine the degree of digital divide among different segments of societies around the world and discuss strategies for minimizing its effect on interpretation of assessment results.Our research focuses on gender and socioeconomic status (SES) groups, two types of demographic groups for whom digital divide has been demonstrated by previous research.We examine digital divide in 21 jurisdictions based on data from the International Computer and Information Literacy Study (ICILS) conducted in 2013.Findings from this research provide insights about the degree and nature of differences, and the factors associated with these differences that may inform policy in addressing inequities.In the final part of this article, we discuss strategies to address digital divide in educational assessments.

What is the Digital Divide?
Digital divide is defined as a social inequity between individuals regarding (1) access to information and communication technology, (2) frequency of use of technology, and (3) ability to use Information Computing Technology (ICT) for different purposes (Hohlfeld, Ritzhaupt, Barron, & Kemker, 2008).There is consistent evidence of differences in ICT access, use and skills between gender, socio-economic status (SES) and ethnic groups.In previous research, an association between digital skills and home ICT access, SES, gender, and history of using ICTs was identified (Zhong, 2011).Research has also demonstrated gender differences with evidence of boys having better technology skills and more positive attitudes toward computers than girls have (Hargittai & Shafer, 2006;Imhof, Vollmeyer & Beierlein, 2007).In addition, poor and minority families in the USA have been identified as being less likely to have access to a computer and broadband Internet connection at home and less likely to have the necessary skills and knowledge to meaningfully use these resources (Attewell, 2001;Hesseldahl, 2008).There also is evidence that African-American males are less likely to meaningfully use ICT resources when compared to their African-American female counterparts as well as both male and female Caucasians (Jackson et al., 2008).
As paper-and-pencil assessments transition into digital environments, examinee ICT capabilities become relevant to their performances on assessments.The ICT-related knowledge, skills and competencies may affect students' ability to read, write, navigate through and engage with the assessments digitally.As demonstrated by previous research, ICT usage is closely related to academic performance in general (Jackson et al., 2008).Even though many assessments transitioning to digital environments are conducting mode effect studies, examining comparability of scores from paper-based versus digitally based assessments, digital divide may affect the validity of comparisons of gender, SES, and ethnic group performances.When groups have different access to and experience with technology, mode effects may be different for these groups, which may affect score comparisons.That is gender, ethnic and language group score comparisons may be functions of digital divide and not reflect true group differences.

Data Sources and Methods
ICILS, conducted by the International Association for Assessment of Educational Achievement (IEA) in 2013, assessed 14-year-olds' Computer and Information Literacy (CIL) in 21 jurisdictions that spanned Europe, Latin America and Asia.Most of these jurisdictions were countries but also included two provinces (Ontario and Newfoundland/Labrador) in Canada and a city in Argentina, Buenos Aires (see Table 1).

Table 1
Participating jurisdictions (sample sizes ranging between 1,000 to 3,700) *did not meet sampling requirements The target construct of the CIL assessment is defined as an "individual's ability to use computers to investigate, create, and communicate in order to participate effectively at home, at school, in the workplace, and in society" (Fraillon, Schulz & Ainley, 2013).The CIL consists of two strands: (1) collecting and managing information and (2) producing and exchanging information.The Collecting and managing information strand assesses students' knowledge about and understanding of computer use, ability to manage information and their ability to access and evaluate information.The Producing and exchanging information strand assesses students' ability to create, transform, share and use information safely and securely in digital environments.There are 62 tasks, organized into four modules.Each module is organized around a theme that included setting up an online collaborative workspace, planning a website, managing files and using online database tools.Each student was administered two of the four modules (See Figure 1).Responses to the tasks were scaled using item response theory and CIL scores were set to have a mean of 500 and standard deviation of 100.

Figure 1. Description of the four modules
Our research examined the degree of differences between gender and SES groups in CIL as well as factors that might be associated with the group differences.The CIL scores were compared for gender and the three SES groups (lowest, medium, highest).This was followed up with exploration of key variables that may be associated with digital divide between gender and SES groups and included student access, interest and experience.Gender and SES groups were compared based on their responses to the following questions and the composite scale scores based on responses to a set of questions related to the themes:  Number of computers currently used in the home  Interest and enjoyment using ICT (composite scale)  Use of specific ICT applications (composite scale)  Use of ICT for social communication (composite scale)  Use of information for exchanging information (composite scale)  Use of ICT during lessons at school (composite scale).

After School Exercise
Students set up an online collaborative workspace to share information and then selected and adapted information to create an advertising poster for an after-school exercise program.

Band Competition
Students planned a website, edited an image, and used a simple website builder to create a webpage containing information about a school band competition.

Breathing
Students managed files and collected and evaluated information needed to create a presentation explaining the process of breathing to 8-or 9year old students.

School Trip
Students helped plan a school trip using online database tools.Students required to select and adapt information to produce an information sheet about the trip for their peers.

Digital Divide in 21 Jurisdictions
The digital literacy skills and competencies are fundamental for learning, employment and daily functioning of individuals and have great importance for educational systems around the world.Additionally, differences among jurisdictions with respect to ICT skills can have an impact on the comparability of jurisdiction performances on international assessments administered digitally.Before delving into the digital divide within jurisdictions we present a brief overview of differences across jurisdictions.The scores of students in 21 jurisdictions who participated in ICILS are summarized in Figure 2 below, ordered from highest (Czech Republic) to lowest (Turkey).Results indicate a great degree of differences in CIL scores between jurisdictions.Two of the jurisdictions, Thailand and Turkey, on the average performed more than one standard deviation below the international average of 500 (see Figure 2).

Gender Differences
Results indicate a large within jurisdiction variation on CIL scores in addition to between jurisdictions.In all jurisdictions, except Buenos Aires, Switzerland, Thailand, and Turkey, differences between gender groups are large in favor of females (Figure 3).This finding is a reversal of the gender differences in digital literacy and technology skills that were identified in previous research.In Buenos Aires, Switzerland, Thailand, and Turkey, the observed differences were also higher for females, but they were small and statistically not significant.In addition to presenting gender differences, Figure 3 highlights within jurisdiction variability in scores that complicate simplistic interpretation of country rankings.For example, even though the Czech Republic has the highest CIL scores overall, girls in Ontario, Australia, Norway and Korea are scoring as high as the Czech boys.We explored how gender groups varied with respect to access, interest and enjoyment, and experience with ICT.Results indicated significant differences between gender groups on the number of computers used at home, interest in and enjoyment of ICT, and use of ICT for social communication.Consistently across all jurisdictions, boys reported having greater access to computers at home (See Figure 4), and in many cases, these differences were statistically significant.This is a surprising finding given that on average the gender groups come from comparable SES backgrounds and therefore from families with similar levels of affluence.Additionally, girls tend to be outperforming boys despite reporting lower levels of access to computers at home.This finding could be due to two possibilities: (1) Reporting bias.That is boys could be reporting more, or girls less, access to computers in households even though access to computers may be similar; (2) Parents may have gone to greater effort to provide computer access for boys than for girls; such systematic bias may tell us something of social norms regarding gendered differences in computer access.For instance, parents may think is is more important to obtain a computer for boys to play games on than for girls.Interest and enjoyment typically were among the strongest predictors of learning and achievement.Despite performing lower compared to girls, boys tended to report greater interest and enjoyment in ICT in all jurisdictions (see Figure 5).This finding raises questions about whether boys and girls use ICT differently.Indeed, girls reported greater use of ICT for social communication in all the jurisdictions except in Turkey, where such use of ICT was higher for boys (see Figure 6).Girls also reported greater use of ICT for exchanging information in some jurisdictions; however, patterns of differences were inconsistent across jurisdictions (Figure 7).Although girls reported greater levels of use of ICT for exchanging information in Ontario, Australia and Newfoundland and Labrador, boys reported higher use of ICT for such purposes in the Czech Republic.No significant differences were observed between boys and girls in use of ICT applications, except in Turkey where boys report greater use of ICT applications and use of ICT in lessons at school.

SES Differences
It is important to keep in mind that on average gender groups are expected to have comparable access to technology at home and school yet are demonstrating large differences in performance on CIL in a great majority of the jurisdictions.As segments of the society, SES groups are the ones for whom we would expect the largest digital divide.We have used the composite SES scale available in the ICILS database.SES in ICILS (S_NISB) is a composite scale with a mean of 50 and standard deviation of 10, based on student-reported parent occupations and educational attainment, and home literacy resources.This composite scale was used to split students into three SES groups within each jurisdiction based on within country distributions: low (lowest 33%), medium, high (highest 33%).
The findings indicate a SES digital divide in all jurisdictions except for the Netherlands, where SES data did not exist (see Figure 8).The gap is lowest in Hong Kong (24 scale score points between lowest and highest SES) and highest in Thailand (93 scale score points between lowest and highest SES).These differences range between half a standard deviation in Hong Kong to almost a full standard deviation in Thailand.It is important to note that the highest SES groups in all the jurisdictions performed well, above the international mean, except in Thailand and Turkey.

Figure 8. Jurisdiction mean CIL scores by SES with 95% confidence interval
Similar to gender group analyses, we examined access, interest and use by SES in each jurisdiction.The results indicated significant differences between SES groups on the numbers of computers used at home, interest and enjoyment using ICT, use of specific ICT applications, and frequency of use of computers at home.
As expected, the findings indicate significant differences in the number of computers used at home between the lowest and the highest SES groups, except in Norway and Korea (see Figure 9), with highest SES groups reporting use of approximately 0.5 computer higher than the lowest SES group.Large differences could be seen in some of the high performing jurisdictions such as Ontario, Australia and Switzerland as well as the low performing jurisdictions such as Thailand and Turkey.Unlike differences in access to computers, differences in interest and enjoyment were small and statistically insignificant between SES groups except for Hong Kong, Chile, Thailand and Turkey, where mixed results were obtained (Figure 10).Although in Hong Kong, Thailand, and Turkey the highest SES groups reported greatest interest and enjoyment, in Chile the lowest SES group reported higher interest and enjoyment than the highest SES group.This finding in Chile is consistent with the gender differences in interest and enjoyment where boys performed lower despite reporting greater levels of interest and enjoyment in ICT.Similarly, in Chile, the lowest SES group performed much lower than the highest SES group (close to one standard deviation on the CIL scale) yet are reporting greater levels of interest and enjoyment.Both gender and SES group trends in Chile suggest a weak link between reported interest and enjoyment of using ICT with performance on the more complex CIL tasks.For example, playing games on computers may not indicate competency in solving complex CIL tasks and enjoying doing that does not translate into competencies associated with CIL.The differences between SES groups were also reported in students' use of ICT applications, for social communication and for exchanging information.However, there was a not simple pattern for SES differences across jurisdictions.In all jurisdictions students from the highest SES groups reported higher levels of use of ICT applications (See Figure 11), except for Chile and Buenos Aires where the differences were small.Patterns for use of ICT for social communication varied across jurisdictions.In high performing jurisdictions such as Czech Republic, Ontario, and Australia the highest SES groups reported the lowest level of use of ICT for social communication, whereas in low performing jurisdictions the highest SES groups used ICT for social communication the most (Figure 12).In most jurisdictions there were very small differences between SES groups in their use of ICT for social purposes; the exceptions were in Thailand and Turkey.

Figure 12. Use of ICT for Social Communication by SES with 95% confidence intervals
Use of ICT for exchanging information patterns varied similarly.The patterns were consistent to some degree with use of ICT for social communication.In high performing jurisdictions the lowest SES group reported the greatest usage, whereas in the lowest performing jurisdictions the opposite was true.However, the order of the use of ICT for exchanging information did not have a consistent pattern for SES groups among the rest of the jurisdictions (Figure 13).The differences between SES groups described so far have been primarily in out-of-school contexts.An important question is whether students from different SES groups have different experiences with computers within school contexts, given they are likely to attend schools with differing degrees of ICT resources.In the ICILS, with respect to use of ICT during lessons at schools, we see variability across jurisdictions but not SES (See Figure 14).There does not seem to be a correlation between use of ICT in lessons and CIL scores.In the highest performing jurisdiction, the Czech Republic, for example, students have reported lower levels of use of ICT in lessons than two of the lowest performing jurisdictions, Thailand and Turkey.

Summary and Implications
There is consistent evidence of gender as well as SES digital divide with respect to access, experience and use in the 21 jurisdictions we examined.The gender gap in scores tends to range between 13 scale score points in the Russian Federation to 38 scale score points in Korea in favor of girls, among jurisdictions where significant differences were observed (Buenos Aires, Switzerland, Thailand, and Turkey showed no significant gender gap).The SES gap is larger, ranging between 24 scale score points in Hong Kong and 93 scale score points in Thailand.Even though the focus of this chapter is on digital divide within jurisdictions, between jurisdictions differences in ICT skills can have important implications for comparing these jurisdictions on international assessments more broadly.As mentioned earlier, this is important where international assessments increasingly may be taking place using digital electronic means.It is important to note that the largest differences in CIL scores were observed between rather than within jurisdictions.Even though 16 of the 21 jurisdictions had average scores above the international mean of 500, the five lowest performing jurisdictions had average scores that were at least 100 scale score points, or 1 standard deviation, lower than the highest performing jurisdiction, the Czech Republic.The two lowest performing jurisdictions Thailand (average 373) and Turkey (average 360) had average scores that were close to 1.5 standard deviations lower than the Czech Republic (average 553).
These findings have important implications for education systems, particularly those for whom educational assessment and evaluations are conducted in digital environments.The results regarding SES differences confirm previous findings regarding SES digital divide; however, gender differences are reversed in the ICILS 2013 compared to previous research, where boys tended to outperform girls.Both sets of findings highlight important implications for education systems.In the paragraphs below, we discuss considerations and strategies for addressing the digital divide in educational assessments in order to minimize the effects of digital divide for different groups.
Educational assessments are used to examine trends in achievement gaps and compare student performance between ethnic, language and other groups over time.The existence of a digital divide with respect to CIL between jurisdictions and between gender and SES groups within jurisdictions points to a possible widening achievement gaps on assessments which may not be true reflections of group differences in knowledge, skills and competencies, but rather reflect differences in ICT knowledge, skills and competencies.As we move from paper and pencil assessments to digital assessments, what can we do to minimize effects of the digital divide on assessment results?We recommend that the following six strategies be used in combination with each other.
1. Familiarize students with the digital environment and assessment mode as part of assessment, as well as prior to assessment, by distributing sample tasks to schools and classrooms.These practice tasks/tests can be used by teachers and educators in general in classroom contexts and can play an important role in providing opportunities to students who may not be familiar with task formats or the digital platform to engage with them and minimize potential effects of digital divide.
2. Provide widely accessible, effective tutorials prior to or during the assessment.A tutorial about the assessment that describes the assessment as well as introducing students to how to navigate when taking the assessment, how to use different tools, and how to enter their responses can play an important role in reducing the disadvantage some students may have due to unfamiliarity with the digital test-taking experience.
3. Administer surveys of digital access and experience along with assessments to examine and understand the degree to which ICT skills are related to performance on the assessment.It is unlikely to be able to eliminate all disadvantage the digital divide may have on assessment performance.An important step to examining the effect of digital divide on the validity of inferences can be explored by examining the degree to which familiarity with the performance is associated with performance on the assessment.
4. Design tests with examinees with the least access and experience in mind instead of taking advantage of the most recent technological developments.Finding the right balance between developing digital assessments that students are currently familiar with and ones that will continue to be appropriate several years into the future when assessments may still need to be used is indeed challenging.However, designing assessments with the most advanced technology is likely to disadvantage students who may not have had a chance to use such technology.
5. Conduct try-outs, think aloud protocol studies or cognitive labs to examine student engagement with digitally-based assessments.Even though the design principles above may provide good rules of thumb, effects of each strategy on performance or effects of different technological elements are not known.Small scale, targeted try-outs, think-aloud studies, or cognitive labs conducted with diverse student groups with differing degrees of ICT skills can help identify which design elements may need to be altered to minimize digital divide effects.
6. Continue checking for possible digital divide and mode effects over time as they are expected to change with the changing access and experience with ICT.We can be certain to expect shifts in both the levels of ICT competencies as well as the degree and nature of digital divide, in similar ways as the gender differences have reversed during the last decade, with girls now outperforming boys.What may seem cutting-edge technology available to only the most advantaged students may become commonplace and widely available in a short period of time, as we have experienced during the last 20 years.Consequently, any disadvantage due to digital divide may change both in nature and size.Therefore, the appropriateness of the digital environment and its potential effect on student performance needs to be examined and updated as part of validity investigations.

Figure 3 .
Figure 3. Jurisdiction mean CIL Scores by gender with 95% confidence interval

Figure 4 .
Figure 4. Number of computers used at home by gender with 95% confidence intervals

Figure 5 .
Figure 5. Interest and enjoyment in using ICT by gender with 95% confidence intervals

Figure 9 .
Figure 9. Number of computers used at home by SES

Figure 13 .
Figure 13.Use of ICT for exchanging information by SES

the Guest Editors Lorin W. Anderson University
in large-scale assessments of educational outcomes and research methods in education.She has conducted research on translation, language and cultural issues in measurement, validating score meaning using response processes, assessment of historical thinking, and the contribution of different research paradigms to creating knowledge and making generalizations in education research.In 2000, she received an Early Career Award from the University of British Columbia, and in 2010, she received the AERA Division D Significant Contributions to Educational Measurement and Research Methodology Award for her co-edited volume Generalizing from Educational Research: Beyond Qualitative and Quantitative Polarization.She has been a member of the National Academy of Education Committee on Foundations of Educational Measurement and has served as an elected member of the National Council on Measurement in Education Board of Directors.She is currently a Fellow of the International Academy of Education and Vice-President of AERA's Division D. Mustafa Asil University of Otago Mustafa Asil is a research fellow at the Educational Assessment Research Unit (EARU), University of Otago, New Zealand.He is responsible for providing research and psychometric support in the execution of the National Monitoring Study of Student Achievement (NMSSA) Project.Mustafa is a psychometrician and a quantitative data analyst with a strong research interest in comparability of large-scale assessments across languages and cultures, and measurement equivalence.Grover Ministry of Education, British Columbia, Canada Raman Grover is a psychometrician at the Ministry of Education for the province of British Columbia in Canada.He provides psychometric expertise for large-scale provincial assessment programs.Raman's research interests are focused on issues of fairness and bias for disadvantaged populations taking multilingual assessments, including investigating score comparability and differential item functioning in heterogeneously diverse testing populations. of South Carolina (Emeritus) anderson.lorinw@gmail.comLorin W. Anderson is a Carolina Distinguished Professor Emeritus at the University of South Carolina, where he served on the faculty from August, 1973, until his retirement in August, 2006.During his tenure at the University he taught graduate courses in research design, classroom assessment, curriculum studies, and teacher effectiveness.He received his Ph.D. in Measurement, Evaluation, and Statistical Analysis from the University of Chicago, where he was a student of Benjamin S. Bloom.He holds a master's degree from the University of Minnesota and a bachelor's degree from Macalester College.Professor Anderson has authored and/or edited 18 books and has had 40 journal articles published.His most recognized and impactful works are Increasing Teacher Effectiveness, Second Edition, published by UNESCO in 2004, and A Taxonomy of Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, published by Pearson in 2001.He is a co-founder of the Center of Excellence for Preparing Teachers of Children of Poverty, which is celebrating its 14 th anniversary this year.In addition, he has established a scholarship program for first-generation college students who plan to become teachers.