Investigating Student Exposure to Competency-Based Education

In recent years, most U.S. states have revised policy by providing schools at least some flexibility to move away from the Carnegie unit system, with its focus on credits and “seat time,” toward competency-based policies that link student advancement to mastery of content. Yet, there is little systematically collected information about how competency-based education is implemented, making it difficult to evaluate the impact on student outcomes. Using data from 600 students in grades 9-12 and confirmatory factor analytic techniques, we report initial reliability and validity results from the pilot administration of a survey designed to capture student exposure to elements that have been described as essential to a competency-based, student-centered model for learning and instruction. These elements include mastery-based progression, personalization, flexible assessment, and the development of specific skills and dispositions. Results suggest that the survey offers a way to reliably measure and study variation in the implementation of competency-based education. Importantly, the survey provides a way to capture implementation from the student perspective, leveraging the fact that student reports about their classroom experiences may be a particularly reliable source of information about instructional practice.


Introduction
Many states are beginning to move away from policies under which student advancement is based on the traditional Carnegie unit system, with its focus on credits and "seat time," and toward policies that provide schools with the flexibility to link advancement to a student's mastery of content (Marion & Leather, 2015;Scheopner-Torres, Brett, & Cox, 2015).The movement toward competency-based approaches to learning and instruction addresses a recommendation from the National Center on Time & Learning (2011) that learning programs should factor in the different amounts of time students need to achieve proficiency and should consider basing student advancement on proficiency, rather than on the amount of time students spend sitting in a classroom.This shift also reflects the position that students are more engaged in their learning when they are granted greater flexibility in how they earn credit (United States Department of Education, 2011).Stakeholders representing a variety of perspectives are increasingly embracing the notion that student-centered (Jobs For the Future, 2014) reforms like competency-based education will yield increased achievement and better preparation for college and the workforce.Similar claims have been made about other so-called student-centered approaches, including both personalized learning (Bill & Melinda Gates Foundation, 2015) and deeper learning (Alliance for Excellent Education, 2012).
Both educators and researchers report that implementation of competency-based education (also referred to as proficiency-based education, or simply competency education) varies widely (Scheopner-Torres et al., 2015;Steele et al., 2014).Implementation challenges include carrying out necessary yet substantial shifts in instruction, assessment, and grading in the classroom; addressing student misconceptions about such shifts in school practices and policies; and supporting student development of the skills and dispositions necessary for student success under a competency-based model (Le, Wolfe, & Steinberg, 2014;Scheopner-Torres et al., 2015).Further, there is a dearth of systematically collected information about how and to what extent instructional practices reflecting elements described as essential to competency-based education are being implemented, making it difficult to investigate how competency-based reforms are or are not leading to changes in student outcomes (Freeland, 2014;Haynes et al., 2016;Le et al., 2014).Moreover, and perhaps surprisingly given the emphasis on student voice in the literature on competency-based education (Le et al., 2014), relatively little is known about how students perceive and experience changes to school policies and instructional practices.
In what follows, we report results from the pilot administration of a survey designed to capture student exposure to elements currently described as essential to competency-based education.These elements, which are multi-dimensional, include mastery-based progression, personalization, flexible assessment, and the development of specific skills and dispositions (Sturgis, 2016).Valid and reliable measures of student exposure to elements of competency-based education are needed if researchers are to more carefully investigate the relationship between implementation and student outcomes.Given that student reports about their classroom experiences appear to be a reliable source of information about instructional practice (Kane & Staiger, 2012), measures based on student-reported exposure to competency-based education hold promise for facilitating research of this nature.The results indicate that this survey reliably measures various dimensions underlying each of the hypothesized elements of competency-based education, providing support for the future use of the survey by both researchers and practitioners to examine implementation and its association with student outcomes.Yet, the results also raise questions about whether competencybased education is a distinct, bounded model for secondary education comprising all four elements described as essential to the approach.We conclude by discussing the implications of these findings for practice, for research, and for policy.

Renewed Interest in Competency-Based Education
In recent years, competency-based education has garnered renewed attention as a reform to increase graduation rates and to ensure that students are prepared for postsecondary success (Scheopner-Torres et al., 2015).The theory is that a larger share of students will ultimately reach proficiency in a given content area if they are given the freedom to advance at their own pace and if their learning experiences are tailored to their needs and interests (Lewis et al., 2014;Sturgis & Patrick, 2010).
Several generations of competency-based education have been documented in the research literature, tracing back to at least the 1960s (Brown, 1994).Current interest in this reform continues a long tradition of challenging the idea that learning happens at a particular pace and place for all students (Brown, 1994;Dewey, 1938;Le et al., 2014;Spady & Mitchell, 1977).Until recently, however, competency-based education was primarily considered a niche approach targeting vocational education and the adult learning segment of the higher education system (Ford, 2014).Recent advances in online learning, learning analytics, and adaptive learning technology, combined with growing demand for demonstrable college and career readiness outcomes among high school graduates, have prompted renewed and expanded interest in the development of major competencybased initiatives (Ford, 2014).Marking a shift from earlier iterations of competency-based education, policymakers have been more responsive to the present movement (Steele et al., 2014).As of 2014, 42 states had authorized at least some flexibility for local education agencies to link the completion of academic credit to demonstration of proficiency rather than only to seat time (Carnegie Foundation for the Advancement of Teaching, 2014).
Over time, competency-based education has been described using a variety of frameworks, all of which share some degree of overlap.Several decades ago, Spady (1977) described competencybased education as consisting of six critical elements including outcomes, time, instruction, measurement, certification, and program adaptability.More recently, based on interviews with educators and a literature review of recent developments in the competency-based education movement, Steele and colleagues (2014) articulated three core principles, including flexible pacing, opportunities to personalize learning, and the requirement that students demonstrate proficiency (and earn credit) by applying knowledge and skills.Most recently, Sturgis (2016) outlined the following four elements as essential to the approach: student advancement through demonstration of mastery based on explicit and measurable learning objectives reflecting important competencies; personalized instruction that provides a student with customized supports and opportunities to engage in anytime/anywhere learning; multiple modes of, and multiple opportunities for, assessment that allow a student to apply his or her learning; and an emphasis on the development of specific skills and dispositions critical for success in a learning environment where students are expected to play a more prominent role in directing their own learning.The research described below, and the student survey used in the research, focus on the four essential elements described by Sturgis (2016).These elements share substantial overlap with the components of competency-based education articulated both in other recent literature (for example, see Competency Works, 2014;Patrick & Sturgis, 2013;Scheopner-Torres et al., 2015;Steele et al., 2014) and in legislation and policy in states that have adopted competency-based education.
Alongside renewed interest in the promise of competency-based education has been a "wild proliferation" (Weise, 2014) of interest in student-centered reforms broadly, perhaps especially in the realm of education philanthropy.For example, the past several years have also witnessed growing enthusiasm for other student-centered models, including personalized learning (Bill & Melinda Gates Foundation, 2015) and deeper learning (Alliance for Excellent Education, 2012;William & Flora Hewlett Foundation, 2014), as a path to better student college and workforce preparation.While the terms used to name these overlapping approaches vary, they share in common some or all of the elements used to describe competency-based education.This suggests growing consensus about the core of student-centered educational reform broadly, and the centrality of these elements to student-centered reforms.Nonetheless, the variation in terminology is noteworthy, particularly to the extent that it both reflects and contributes to imprecision, if not confusion, in how the elements of competency-based education and other student-centered approaches are defined-an issue that is not new, but was recognized some three decades ago by Spady and colleagues (Spady, 1977;Spady & Mitchell, 1977).Echoing this lack of precision, both in language and meaning, is wide variability within and across schools and districts in how competencybased policies and practices are conceived and implemented.Ultimately, these issues pose significant challenges for efforts to understand the relationship between the implementation of competencybased education and student outcomes.

Implementation of Competency-Based Education and Student Outcomes
While many are enthusiastic about the promise of competency-based education, the bridge between a promising idea and impact on student outcomes is implementation-and innovations are seldom implemented as intended (Berman & McLaughlin, 1976).Although there is general agreement among proponents of competency-based education about its essential elements, implementation varies substantially across states and districts (Steele et al., 2014;Scheopner-Torres et al., 2015;Stump & Silvernail, 2015).Competency-based policies range from those that simply allow flexibility in awarding credit, to policies that completely transform the education system "from a time-based system to a learning-based system" (Sturgis & Patrick, 2010, p. 4).Some states require implementation of a competency-based diploma system in all districts; in other states districts can choose to implement elements of competency-based education, but state policies neither require nor restrict implementation of the reform.
States also vary in how they implement competency-based grading policies and graduation requirements (Sturgis, 2014).In some contexts, districts are required to make competency-based credits available, but in other places districts are allowed to decide whether these will be offered.Some states specify how students will demonstrate competency (e.g., scores on statewide or national tests, portfolios), while others leave it to the discretion of the district to determine how competencies are measured.Certain states allow credits to be awarded for demonstration of competency in any course, but others only offer this in certain subjects.
Beyond variation in the nature of state competency-based policies, how these policies are implemented at the local level also differs.For instance, one study (Steele et al., 2014) of implementation in several districts throughout the United States documented variability in which competency-based education elements districts chose to implement.Some districts focused primarily on the assessment of proficiency on competencies, but others focused on personalization of learning, including student choice and flexible pacing.This study also found variability across sites in a number of other areas including: the extent to which learning experiences that occurred outside of school hours and off of school grounds were counted toward the fulfillment of course requirements, the criteria used to determine student proficiency on a specific standard or competency, and the use of technology to facilitate online learning opportunities and operate learning management systems.
In order to develop a body of empirical evidence about how competency-based education is related to student outcomes, it will be necessary to continue building a better understanding of variability in implementation, including the development of valid and reliable ways to measure implementation fidelity.In particular, practitioners and policymakers would benefit from a better understanding of how students' experiences with competency-based education are consistent with current descriptions of the model and its component elements.As research about the use of student perception surveys to gather information about students' classroom experiences has demonstrated, student surveys may provide more consistent results than classroom observations about what is happening in the classroom (Balch, 2012;Ferguson, 2012;Kane & Staiger, 2012), offering an important source of information about implementation.
Researchers describe five aspects of implementation fidelity (Dane & Schneider, 1998;Durlak & DuPre, 2008;Dusenbury, Brannigan, Falco, & Hansen, 2003).Adherence references whether specific features of a program are implemented as prescribed by the program model.Exposure reflects the amount of the program delivered and/or the extent to which those served by the program were exposed to each element of the model.Quality refers to how well the program is implemented, including the caliber of the delivered model features.Program differentiation indicates the degree to which the essential elements of a program are distinguishable from each other and from other programs.Finally, responsiveness reflects the manner in which those exposed react to or engage in the program.
At the student level, the survey used in this research captures information about student exposure to each of the hypothesized elements of competency-based education; this includes information about whether the student is exposed to instructional practices associated with each element and information about the frequency with which the student is exposed to such practices (the survey does not, however, capture duration, or the amount of time a student is exposed to these practices).At the school level, results can provide information about adherence, or the extent to which specific elements prescribed in the current literature about competency-based education are implemented.Results from this survey can also be used to explore program differentiation, or the extent to which hypothesized essential elements of competency-based education are distinguishable from each other and whether, as a whole, these elements constitute a bounded, coherent model that is distinct from other programs-an issue of particular emphasis in this research.

Conceptual Framework
This research offers one of the first empirically based attempts to operationalize and measure what have been proposed as essential elements of competency-based education.Information about students' exposure to competency-based education can support practitioners and policymakers in making more informed decisions about implementation, and can facilitate research addressing the relationship between implementation and student outcomes.
The limited empirical research base on the implementation and outcomes of competencybased education, along with the absence of conceptual clarity about what key elements of competency-based education should look like in practice, pose challenges for measurement.Our approach in the current study is informed by the position that validating new constructs involves an iterative, ongoing process of refining both theory and measurement (Westen & Rosenthal, 2003).We begin this process by drawing from the existing literature on competency-based education to inform the constructs we study in the current research-constructs intended to capture the hypothesized key elements of competency-based education as measured by their underlying dimensions.
At the core of competency-based education is an approach to teaching and learning that allows students to advance in school based on demonstrated mastery of important content, typically as defined through measurable learning objectives reflecting competencies required for success in college, career, and life (Le et al., 2014).One dimension of progression through demonstration of mastery is competency-based pacing (CB PACE; see Figure 1), which involves student exposure to instructional practices that allow them to work at their own level and pace, advancing to the next level of study based on demonstrated mastery of skills rather than on their age or number of hours they've spent in class (Le et al., 2014).A second dimension is competency-specific feedback (CB FEED), which is intended to ensure that students understand what is expected of them, including how their efforts will be assessed and what mastery (or proficiency-the terms are often used interchangeably) looks like.See Appendix Table A1 for a description of each dimension and the indicators of each.
One of the more elusive elements of competency-based education-and other studentcentered reforms-is student personalization.The notion of personalization in K-12 education broadly is both increasingly popular and variably defined (for example, see Le et al., 2014).In the competency-based education literature, dimensions of personalization typically include personalized options for practicing and applying skills (PERS APP; this is sometimes referred to as "anytime, anywhere" learning), along with exposure to personalized instructional practices (PERS INSTR) that respond to the needs of individual students.
With respect to student assessment, the literature on competency-based education also emphasizes flexible assessment (FLEX).Flexible assessment reflects the idea that students should have multiple and varied opportunities to demonstrate mastery of important skills as part of a summative assessment system (i.e., not simply quizzes or exams).An emphasis on flexible assessment also reflects a belief that students may require multiple attempts to demonstrate mastery, such that falling short of mastery on the first attempt is interpreted not as a failure but as indicating a need for additional practice and support before subsequent attempts.
Finally, current literature suggests the need to support student development of specific skills and dispositions hypothesized as necessary for student success within a competency-based framework for learning and instruction.Although we use the phrase skills and dispositions here, we note that the literature uses a range of overlapping terms, including, for example, work-study habits, non-cognitive skills, and socio-emotional skills.Amidst an era of great interest in students' social and emotional well-being, a range of skills and dispositions have been proposed in the literature on competency-based education as important for student success under this model.We focus on five dimensions reflecting the development of student skills and dispositions.The first is exposure to instructional practices that encourage respect for others (RSPCT; Kallick & Costa, 2009).The second is exposure to instructional practices that encourage students to persevere (PRSVR; Farrington et al., 2012) even when faced with a challenge.We also focus on exposure to instructional practices that encourage students to take responsibility for their learning (RSPNS; Lewis et al., 2014), as well as student demonstration of ownership over their learning (OWN; Le et al., 2014).Finally, we focus on student exposure to instructional practices that encourage peer collaboration (COLL; Kallick & Costa, 2009).
In this research, we model latent factors representing student exposure to the dimensions underlying hypothesized elements of competency-based education.Using data collected via a survey designed to capture student feedback about the implementation of competency-based education in their courses, we address three research questions: (1) Can dimensions underlying hypothesized elements of competency-based education be reliably measured?; (2) To the extent that dimensions of competency-based education elements can be reliably measured, how are they associated with one another?; and (3) To what extent do the results provide evidence of construct validity?
We investigate these questions using confirmatory factor analysis (CFA), which can show whether the observed variables hypothesized to measure an unobserved construct demonstrate a level of shared variance high enough to suggest that those items represent a common underlying factor (i.e., the construct of interest).Per the conceptual framework depicted in Figure 1, we consider three potential ways of representing student exposure to dimensions underlying the hypothesized elements of competency-based education.We begin with a correlated traits model (Model A) in which various dimensions of instructional practice underlying the four key elements of competency-based education are represented by the 10 latent factors depicted in Figure 1.In the correlated traits model, the 10 latent factors are considered to be related to varying degrees but no attempt is made to measure one or more higher-order factors.  1 and Appendix Table A1), which are not depicted.
To explore whether dimensions underlying the hypothesized essential elements of competency-based education can be represented in a manner consistent with the current description of competency-based education as a distinct program model, we also consider 2 second-order factor models.In Model B (see Figure 1), which is most consistent with current descriptions, the dimensions were grouped as indicators of second-order factors representing student exposure to the hypothesized key elements of competency-based education.This model assumes that the secondorder factors (elements of competency-based education) explain the correlations among the firstorder factors (dimensions of each element).We also explore an alternative model (Model C; see Figure 1) that assumes the correlations among the 10 first-order factors are explained by one overall competency-based education second-order factor rather than by a set of component elements.

Method Participants
After securing Institutional Review Board approval, the data for this study were collected in the spring of 2016 at two high schools in the Northeast, including a small rural high school and a midsize suburban high school.Approximately 80% of students in grades 9-12 (n=599) at the two schools completed the survey; missing responses primarily reflected student absence on the date of survey administration (n=83), including a field trip for approximately half of students in grade 12 at one high school.Students in grade 9 made up 33% of the sample, and students in grade 10 made up 24%, followed by students in grades 11 (29%) and 12 (14%).Just over half (52%) of the sample was female.A majority of the sample expected to complete a bachelor's degree (39%) or above (42%).

Survey Development
We developed this survey in the context of a research-practice partnership that included teachers, school and district administrators, state department of education staff, and researchers.The work of the partnership, including the development of the survey, was funded by the Institute of Education Sciences through the Regional Educational Laboratory (REL) program and was facilitated by researchers at REL Northeast and Islands, one of 10 RELs across the country.The overall goal of the REL partnerships is to support practitioners' and policymakers' use of empirical evidence to inform policy and practice and to improve student outcomes.Partnership research is guided by the needs of these stakeholders who, in the case of this survey, identified the need to capture student feedback about how competency-based education is actually being implemented.Resources developed through the partnership are freely available and no organization or individual profits from their use.The survey used in this research, the Competency-Based Learning Survey for Students (Ryan & Cox, 2016), can found at https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=REL2016165.
We used seminal survey development resources, including Fowler (2008), Rea and Parker (2005), and Wright and Marsden (2010), to guide the development of the survey instrument.Survey development relied on close collaboration between the researchers and a six-member advisory committee made up of practitioners from schools implementing competency-based education along with researchers having expertise in competency-based approaches to learning and instruction at the secondary level.Ongoing collaboration with the advisory committee was critical to ensure that the survey items would elicit useful, valid, and reliable information about student exposure to competency-based education policies and practices.Practitioner involvement was especially valuable for ensuring that the item language would be meaningful for students across a variety of state contexts, particularly given variability in terminology surrounding competency-based education.
Several steps were taken to establish face (i.e., Does the instrument measure what it is intended to measure?) and content (i.e., Does the instrument tap into the various dimensions of the specific constructs of interest?)validity.First, in collaboration with the advisory committee, we developed an initial pool of survey items aimed at providing complete coverage of the hypothesized constructs.The initial item pool included more items than were necessary to ensure complete coverage.The advisors reviewed all draft items, providing feedback on each item in response to a number of specific prompts.This process was used to evaluate the items' face validity and content validity.We refined the item pool in response to this feedback, and we cycled through this iterative process of review and revision until the item pool covered the constructs completely and constituted a survey that could be administered without overburdening the survey respondents or the school staff that would administer the survey.The refined item pool formed the first draft of the survey.
We then administered the initial survey draft to seven students and conducted cognitive interviews with these students.Cognitive interviewing is a method for identifying and correcting problems with surveys that involves administering a draft survey to a respondent while concurrently interviewing him or her to determine if the survey items are eliciting the information that the researcher intends (Beatty & Willis, 2007).During cognitive interviews respondents were asked to think aloud as they completed the survey, thereby making explicit their interpretations of instructions, items, and response choices, as well as their decision points.Information gathered during the cognitive interviews was analyzed and discussed with the advisory committee to improve the clarity, relevance, length, and coverage of survey items.

Measures
We describe below each of the 10 latent factors we modeled to represent different dimensions of the hypothesized elements of competency-based education.Descriptive statistics for the hypothesized indicators of each latent factor are presented in Table A1.
Mastery-based progression.Two latent factors reflect dimensions of mastery-based progression: competency-based pacing (CB PACE; five items), or exposure to instructional practices associated with the opportunity to progress through demonstration of mastery, and competency-specific feedback (CB FEED; three items), or exposure to guidance and feedback from teachers about how student mastery of important content will be determined.
Personalization.Dimensions of personalization are reflected through two latent factors.The first latent factor reflects exposure to personalized options for practicing and applying skills (PERS APP; five items), and the second reflects exposure to personalized instructional practices based on the needs of individual students (PERS INSTR; six items).
Flexible assessment.The flexible assessment dimension (FLEX; seven items) reflects exposure to multiple and varied opportunities to demonstrate learning.
Skills and dispositions.Five underlying dimensions reflect the skills and dispositions hypothesized as important for student success under competency-based education.These dimensions include: exposure to instructional practices that encourage respect for others (RSPCT; three items); exposure to instructional practices that encourage students to persevere even when faced with a challenge (PRSVR; three items); exposure to instructional practices that encourage students to take responsibility for their learning (RSPNS; three items); student demonstration of ownership over learning (OWN; three items); and exposure to instructional practices that encourage peer collaboration (COLL; three items).

Analysis
We used confirmatory factor analysis (CFA) to examine the fit between each hypothesized model (see Figure 1) and the data.Analyses were conducted using Mplus statistical software, Version 7.0 (Muthén & Muthén, 2016).An advantage of CFA is that the analyses produce a number of fit statistics, making it possible to evaluate how well the hypothesized model fits the observed data.We examined fit indices including the chi-squared statistic, the Comparative Fit Index (CFI) and the Root Mean Square Error of Approximation (RMSEA).While small misspecifications in a model often lead to inflation of the chi-squared statistic even when the model is correctly specified (Curran, West, & Finch, 1996), the CFI accounts for sample size, with a value above 0.90 indicating a good fit between the model and the data (Hu & Bentler, 1999).The RMSEA indicates how well the model would fit the population covariance matrix given unknown but optimally chosen parameters; the suggested upper-limit cutoff is 0.06 (Hu & Bentler, 1999).
The robust weighted least squares (WLSMV) estimator was used to estimate the models described here.The WLSMV estimator is most appropriate in this research given that almost all observed variables are categorical in nature.This estimator uses the probit link and provides standard errors and a chi-squared test statistic that are robust to non-normality.

Dimensional Structure
The initial correlated traits CFA model (Model A [original]; see Figure 1) included 10 latent factors, as described above, each as measured by its originally hypothesized indicators, per Table 1.Results (Table 1) suggested an adequate fit between this model and the data (χ 2 (734)=2181.50;CFI=.92;RMSEA=.06).However, a review of factor loadings revealed three (CB PACE4, PERS INSTR5, and FLEX2) that fell at or below the recommended .40threshold (Stevens, 2009), suggesting that these indicators should be removed from the model.We also reviewed model modification indices, which indicate how the addition or removal of specific parameters would improve model fit.Because modification indices are purely data-based and not informed by theory, over-reliance on modification indices can quickly become an exploratory journey, increasing the risk of Type I error.Thus, no more than a few modifications should be made to an initial confirmatory model, provided that a clear theoretical justification exists (McDonald & Ho, 2002).
The modification indices for the original model included many potential changes.However, we identified just one change as consistent with the literature on competency-based education.The change involved modeling an original indicator (CB PACE1, "I know what I need to do to show my teachers that I am making progress on each competency") of the competency-based pacing latent factor as an indicator of the student demonstration of ownership over learning factor instead.This change is consistent with the suggestion that students must be prepared to guide their own learning under competency-based education (Le et al., 2014;Lewis et al., 2014).In total, four changes were made to the original model: three indicators were dropped from the model, and CB PACE1 became a fourth indicator of student demonstration of ownership over learning (OWN4; see Table 1).This revision to the original model (Table 1, Model A [revised]), which reflects the changes described above (changes in bold font in Table 1), fit the data well (χ 2 (618)=1429.61;CFI=.95;RMSEA=.05).All factor loadings in the revised model were statistically significant at p≤.001.Correlations among the 10 latent factors are provided in Table 2. Our next step was to estimate a second-order factor model, a model in which the first-order latent factors in turn become indicators of a second-order factor.The initial second-order factor model (Model B; see Figure 1) is most consistent with current descriptions of competency-based education as comprising four essential elements.This model included three second-order factors, representing three of the four key elements of competency-based education: mastery-based progression, as indicated by the first-order factors CP PACE and CB FEED; personalization, as indicated by the first-order factors PERS APP and PERS INSTR; and skills and dispositions, as indicated by the first-order factors RSPCT, PRSVR, RSPNS, OWN, and COLL.We did not model the fourth key element of competency-based education, flexible assessment, as a second-order factor given that this element was reflected through just one dimension (FLEX).Model B also fit the data well (χ 2 (648)=1668.91;CFI=.94;RMSEA=.05).Given that Model B is nested within Model A, we used a chi-squared difference test to determine whether Model B fit the data as well as Model A. The result was significant (χ 2diff =227.34,p=.00; see Table 1), suggesting that although both models fit the data well, Model A is preferred over Model B.
We also estimated an alternative second-order factor model (Model C; see Figure 1) in which all 10 first-order latent factors were modeled as indicators of one underlying second-order factor, which we might think of as representing competency-based education.Again, this model fit the data well (χ 2 (653)=1770.28;CFI=.94;RMSEA=.06).Because Model C is also nested within Model A, we again used a chi-squared difference test to determine whether Model C fit the data as well as Model A.Here too, the result was significant (χ 2diff =290.21,p=.00; see Table 1), indicating that Model A is also preferred over Model C. In sum, the CFA results suggest that the correlated traits model (Model A), which included 10 latent factors reflecting different dimensions underlying each of the four elements described as essential to competency-based education, fit the data well.Further, all 10 latent factors were positively and significantly correlated.The second-order factor models (Models B and C), both of which are more consistent with a conceptualization of competency-based education as a bounded, distinct model for secondary schooling as opposed to a more arbitrary mix of various dimensions of instructional practice, also fit the data well.However, neither second-order factor model fit the data as well as the correlated traits model (Model A).We return to the implications of these results in the discussion.

Construct Reliability
To assess the internal consistency of each of the latent factors, we used the reliability coefficient (rho [ρ]) developed by Raykov (2001) and recommended (Brown, 2006;Raykov, 2001) for use in CFA analyses, particularly when factor indicators are categorical.Construct reliabilities above .70are considered to reflect high internal consistency (Kline, 2000).All construct reliabilities in the revised Model A ranged from .74-.85 (see Table 2).

Convergent and Divergent Construct Validity
Within a construct validation framework, convergent validity reflects the notion that measures we would expect to be related are, in fact, related.In contrast, divergent validity reflects the idea that measures we would not expect to be related share only a weak or nonexistent relationship.
Perhaps the most robust evidence of convergent validity is obtained by examining how strongly the measures under study are related to previously established measures of the constructs of interest.Given that there are no existing reliable and valid measures of student exposure to competency-based education, we were not able to examine this kind of evidence.Although we were not able to examine correlations among measures of the same constructs, we did consider the direction and magnitude of the correlations among the latent factors we modeled as another way to explore convergent and divergent validity.Almost all correlations among the 10 latent factors were positive, moderate or strong, and statistically significant (see Table 2).
To explore divergent validity, we examined the correlations between each of the 10 latent factors and two additional survey items (see Table 3): "Students in my courses all work on the same assignment at the same time" and "My teachers spend most of class time giving a lecture or presentation to the whole class."These items were intended to capture the extent to which students reported exposure to practices associated with what is often described as a traditional approach to instruction, or one that is teacher-directed and assumes that students will all move through the same content at about the same pace (Wolfe, Steinberg, & Hoffman, 2013).We anticipated that as student exposure to the various dimensions of competency-based education increased, their exposure to more traditional instructional practices would decrease.Generally consistent with this expectation, all but one of the latent factors shared a negative or nonexistent relationship with both of these survey items.In one exception, the latent factor RSPCT shared a weak positive relationship with the item "Students in my courses all work on the same assignment at the same time."The positive and moderate to strong correlations among most of the latent factors are potentially indicative of convergent validity, or the idea that measures that should be related are actually related.The negative correlations these factors shared with items representing a more traditional approach to teaching and learning are suggestive of divergent validity, or the idea that measures that should not be related are indeed not related.However, we note that these results are exploratory and somewhat difficult to interpret without additional research.

Construct Generalizability
Another aspect of construct validity is generalizability (Messick, 1995).Multigroup CFA can be used to test whether the constructs of interest are generalizable across populations (Sass, 2011) or, in other words, whether the measures under study are invariant across populations.Although this step is often overlooked, it is important to establish measurement invariance (i.e., the same constructs can be measured in the same way across populations) before attempting to draw comparisons across different populations-for example, different schools-on the constructs of interest (Sass, 2011).This research used data from two different high schools, one that was just beginning to implement competency-based learning and one that had been competency-based for several years, providing an opportunity to test whether our measures were invariant across these two groups using multigroup CFA.Muthén and Asparouhov (2002) assert that in order to make meaningful comparisons of factor distributions across groups, a majority of the variables serving as latent factor indicators should have both threshold and loading invariance.The criteria suggested by Muthén and Asparouhov were satisfied (see Appendix for details), indicating that the measures in Model A (revised) were invariant across the two schools.

Discussion
Interest in competency-based and other student-centered reforms at the secondary level continues to grow, but research lags behind (Freeland, 2014;Haynes et al., 2016).As one consequence, changes in educational policy and practice associated with the shift to competencybased education often lack empirical evidence.One obstacle for both researchers and practitioners interested in better understanding student experiences and outcomes under competency-based education has been the lack of tools that could be used to measure and study variation in implementation.Through the testing of a survey that captures student-reported exposure to hypothesized essential elements of competency-based education, this study begins to address the need for measurement instruments that will allow researchers to investigate how this ambitious, if somewhat nebulous, reform is actually being implemented, and with what consequence for student outcomes.
Overall, results from the confirmatory factor analyses indicated that the correlated traits model (Model A, revised) provided the best fit with the data.In this model, various dimensions of the four hypothesized elements of competency-based education were represented by 10 latent factors, all of which demonstrated high internal consistency.The model allowed for correlations among these 10 latent factors, but we did not group them into one or more higher-order factors.We also examined two second-order factor models.The first (Model B) modeled the first-order latent factors as indicators of second-order factors representing elements of competency-based education; the other (Model C) modeled all 10 first-order latent factors as indicators of a single competencybased education second-order factor.The second-order models reflected two ways of representing competency-based education as a bounded, distinct model for secondary schooling, as opposed to a more arbitrary mix of various dimensions of instructional practice (Model A).Although both second-order factors models fit the data well, results from chi-squared difference tests indicated that neither model provided a better fit to the data than Model A. Conceptually, these results indicate that the survey reliably captures distinct dimensions of instructional practice associated with competency-based education.But the results also indicate that these dimensions may not cluster together in a manner consistent with the description of competency-based education as a distinct model that can be distinguished from other approaches to secondary education.We return to this point below in discussing directions for both future research and policy.
With respect to construct validity, we worked closely with a group of six advisors that included practitioners and researchers with expertise in competency-based education to establish both face and content validity of the survey instrument.We also considered the extent to which our results provided preliminary evidence supporting convergent and divergent validity of the constructs operationalized in this research.Without existing reliable and valid measures of student exposure to elements of competency-based education, we were not able to examine how our measures correlated with other established measures, which would have provided the most robust evidence about convergent validity.We did, however, examine the direction and magnitude of correlations among the latent factors we modeled, and almost all of the correlations among the 10 latent factors were positive, moderate or strong, and statistically significant.Nonetheless, we emphasize that these results are exploratory.It is difficult to draw firm conclusions about construct validity, particularly given the limited empirical basis upon which to base expectations about the distinctive elements of competency-based education-including the component dimensions of those elements-and the relationships among them.As we discuss below, future research will be necessary to more comprehensively evaluate convergent and divergent validity and other aspects of construct validity.
For both practitioners and researchers, results from this research suggest that the survey offers a way to reliably measure and study variation in the implementation of practices commonly associated with competency-based education.Importantly, the survey provides a way to capture implementation from the student perspective.In the remaining sections, we outline implications for practice, for future research that can provide needed empirical evidence about a school reform model that is increasingly making its way into state and district policy, and, finally, we discuss implications for policy itself.

Implications for Practice
Given that student reports about their classroom experiences appear to be a particularly reliable source of information about instructional practice (Kane & Staiger, 2012), student-reported exposure to competency-based education may be especially useful as schools seek to understand and refine instruction.Results from student perception surveys such as this one can also provide feedback for the improvement of learning and instruction (Ferguson, 2010), and can be used to inform data-based decision making (Lai & Schildkamp, 2013).For example, where students report low exposure to particular dimensions of instructional practice associated with competency-based education, this could be useful feedback for teachers and schools as they work toward more consistent implementation of related practices.It may also be valuable for practitioners and policymakers to consider differences among particular student subgroups in their reported exposure to dimensions of competency-based education.Although competency-based education has been described as uniquely capable of addressing the academic and social-emotional needs of students who struggle under traditional systems of schooling, whether this is the case remains an open question (Lewis et al., 2014).
Further, either collectively or in collaboration with researchers, practitioners might also triangulate other sources of evidence about instructional practice, for example from classroom observations or staff surveys, with student-reported experiences.This type of analysis would provide better information about adherence, or the extent to which hypothesized elements of competencybased education are implemented as prescribed in the literature at both the classroom and school levels.Such an analysis would also provide information about the extent to which students' perceptions of instructional practices under competency-based education parallel the perceptions of their teachers.Instances of substantial misalignment between student and teacher perceptions about what is happening in the classroom would provide useful opportunities for school staff to reflect together on the sources of this disconnect.Potential sources might include, for example, student and/or teacher over-or underreporting of competency-based instructional practices or nuances in the wording of student survey items.

Implications for Research
With respect to future research, a next step should be to administer this survey across more schools both to assess reliability within schools and to measure variation in how students experience competency-based education among schools and districts.As noted above, subsequent survey administration could be carried out in conjunction with qualitative research, including both observations and interviews in schools, to document the existence of certain practices and compare these data with student feedback.Combining student survey administration with qualitative data collection could also prove valuable for evaluating how much variation in student responses derives from measurement issues as opposed to real variation in their classroom and school experiences.
There is also a need for continued research investigating the program differentiation aspect of implementation fidelity, or the degree to which the essential elements of competency-based education are distinguishable from each other and from other programs.We considered program differentiation in this research by comparing how well three potential ways of modeling competency-based education implementation fit the observed data.The initial second-order model conceptualized the individual dimensions as reflecting underlying elements of competency-based education, and the other second-order model conceptualized these dimensions as reflecting a singular underlying "competency-based education" construct.While both second-order models fit the data well, neither fit the data as well as the initial model in which the dimensions of each element were modeled as correlated but separate.On the one hand, it is possible that we have not fully captured the elements of competency-based education through the dimensions we measure and, were we able to do so, we would find that the separate dimensions (i.e., first-order factors) actually are better modeled as representing one or more of the hypothesized elements (i.e., second-order factors).On the other hand, the results may instead suggest that these dimensions do not cluster together in a manner that supports the conceptualization of competency-based education as a distinct model with four constituent elements.In other words, competency-based education may actually unfold as a more arbitrary mix of instructional practices that, at the level of implementation, are combined in ways that can't necessarily be distinguished from other models for secondary education-a possibility consistent with research indicating that whole-school models often lack the specificity required to achieve, and to measurably detect, high implementation fidelity (Desimone, 2002;O'Donnell, 2008).
Based on these findings, future research on program differentiation might use this survey with students from a larger sample of schools to study whether elements described as essential to competency-based education actually constitute a coherent and distinct model for secondary education.This could include studies that administer the survey both with students in schools that have adopted a competency-based model and with students in schools that have not adopted this approach.Survey administration with both types of students would provide evidence about the extent to which the survey can be used not only to distinguish among schools demonstrating different levels of implementation fidelity, but also to distinguish between schools that have and have not adopted a competency-based approach.Student survey results could also be compared with information collected through other modes (e.g., teacher surveys, classroom observations) in order to examine whether different sources of information about program differentiation, and about other aspects of implementation fidelity, provide similar or different results.
Future research should also continue to explore the validity of constructs reflecting student exposure to hypothesized elements of competency-based education, including the validity of constructs that attempt to combine these elements into one overall measure of competency-based education implementation.In the absence of previously established valid and reliable measures of student exposure to competency-based education, our efforts to assess construct validity were limited to examining the direction and magnitude of correlations among the first-order latent factors as a way to consider convergent validity.Almost all correlations among the 10 latent factors, which reflect different dimensions of the hypothesized competency-based education elements, were positive, moderate or strong, and statistically significant.Yet, we emphasize that these results are exploratory and difficult to interpret without additional research.A correlation between two measures provides a somewhat ambiguous signal about convergent and divergent validity given the potential influences of both trait and method variance (Furr & Bacharach, 2013).A strong correlation may indicate that two measures share trait variance, meaning that the constructs that they are intended to measure share something in common.However, a strong correlation could also indicate that the two measures share method variance, indicating that they are correlated primarily because they use the same method of measurement.Future research using multi-trait, multi-method techniques (Widaman, 1985) could investigate whether factors defined by different measures (e.g., online survey, paper/pencil survey) of the same trait (e.g., dimensions of competency-based education) support the validity of the measures reported here, and also whether factors defined by different traits but measured with the same method argue for method effects.
Ultimately, construct validation involves a perpetual process of refining both theory and measurement (Westen & Rosenthal, 2003).A more comprehensive analysis of construct validity with respect to competency-based education will require continued research, including iterative refinement of both theory and measurement.This should include, for example, collecting longitudinal data to facilitate the investigation of predictive validity, which will also require specificity about the outcomes we should expect to be especially impacted by exposure to competency-based education.At the secondary level, competency-based education is typically proposed as a reform to improve students' college and career readiness-but what does this mean, specifically?For example, is competency-based education intended to improve academic achievement as measured by standardized assessments, performance on college entrance exams, or other measures of college and career readiness, including social and emotional outcomes such as engagement and intrinsic motivation?Or is it intended to impact all of the above?Finally, this survey, which reliably captures student exposure to various dimensions of instructional practice associated with competency-based education, might be used to more carefully investigate the relationship between implementation and student outcomes.A handful of studies in recent years have attempted to estimate the impact of various student-centered approaches, including competency-based, personalized, student-centered, and "deeper" learning approaches (Haynes et al., 2016;Steele et al., 2014;Zeiser, Taylor, Rickles, Garet, & Segeritz, 2014).Yet these studies have yielded limited knowledge about the effects of competency-based education due to considerable variation in its implementation.For example, in one recent study (Haynes et al., 2016), designed to compare outcomes among students in schools identified a priori as competency-based or non-competency-based, the researchers found this to be a problematic distinction.Instructional practices commonly associated with competency-based education in the literature varied substantially within competency-based schools; moreover, these same practices were also evident, and similarly variable, in schools identified as non-competency-based.The researchers concluded that competency-based education cannot be characterized as a present/non-present phenomenon.Rather than treating the implementation of competency-based education as a dichotomous designation, future research might use the survey described in this study to investigate how the nature and extent of student exposure to competency-based education impacts student outcomes.

Implications for Policy
Although policy prescriptions are best informed by a cumulative body of evidence, rather than by any single study (McDonnell, 2000), the results from this research, while preliminary, suggest several directions for policy.
First, we see a need for policy at national, state, and local levels to establish greater precision in the language used to define competency-based education and its key elements.As Hess (2016) notes, the school reform landscape is littered with words that have been stripped of meaning, which is true, to varying degrees, about many of the terms used to define elements of competency-based education, and to define student-centered reforms more generally.Ultimately, the lack of clarity in what such placeholders mean avoids complicated questions (Hess, 2016), including questions about how to coherently define and successfully implement changes in policy and practice, and about how to measure and evaluate these changes and their implications for students.As noted earlier, imprecision in the language used to describe and define competency-based education and related approaches is accompanied by wide variability within and across schools and districts in how competency-based policies and practices are conceived and implemented.Thus, it is perhaps not surprising that our results did not necessarily support the conceptualization of competency-based education as a distinct model with four constituent elements, at least as these elements have currently been defined.
Second, the results, which raise numerous questions for future research, highlight the fact that greater empirical evidence about the nature and effectiveness of competency-based education is needed to inform both policy and advocacy at both the national and state levels.Most U.S. states have now authorized at least some flexibility for local education agencies to link the completion of academic credit to demonstration of proficiency rather than simply to seat time (Carnegie Foundation for the Advancement of Teaching, 2014).Yet, there remains little available empirical evidence about the effect of competency-based education on student outcomes (Freeland, 2014;Haynes et al, 2016;Le et al., 2014).Moving forward, policies at all levels of the education system related to competency-based education would benefit from a stronger grounding in evidence from a variety of sources and using a range of methods.Related, proponents of competency-based and other student-centered educational reforms can best support sound policy by engaging in evidencedriven advocacy-even, or perhaps especially, when the evidence challenges prevailing assumptions about how well these reforms work (McCarthy, 2016).
Third, the results draw attention to the fact that input from students is an important source of information as state and especially district policymakers seek to understand how competencybased education is being implemented in the classroom, and student voices can provide guidance about where changes in policy might be necessary.Recent research about the use of student perception surveys to gather data about students' classroom experiences has demonstrated that student surveys may provide particularly consistent feedback about what is happening in the classroom (Balch, 2012;Ferguson, 2012;Kane & Staiger, 2012), offering an important source of information about implementation.While much of this previous research has been situated in the literature on teacher evaluation, we believe the survey studied in this research is best used not as an accountability tool, but rather as a source of information about where changes in policy may be needed to clarify the defining features of competency-based education and to support implementation.For example, during our discussions with each of the participating districts about their survey results, leaders and teachers identified specific patterns that, from their perspective, indicated a need for both changes in policy and changes in how existing policies were communicated to teachers and students.
Finally, we remind the reader that policy changes, while necessary, are almost certainly insufficient to achieve successful implementation of competency-based education.As Spady and Mitchell (1977) wisely advised several decades ago, doing so will also require attending to the social order of the school, for example by developing a sense of ownership for the proposed implementation among students, staff, and parents, by providing rewards to teachers and students for risking new approaches to their respective roles, and by encouraging greater authenticity in the relationships and goals of students and teachers.

Conclusion
Recent evidence from a nationally representative sample of high school graduates indicates that almost half of U.S. high school students complete neither a college-or career-ready course of study (Bromberg & Theokas, 2016).Policymakers, employers, and the public voice concerns that America's youth are graduating without the skills and knowledge needed for postsecondary success.In response, many states and districts are turning to competency-based education policies and other so-called student-centered reforms as a strategy to ensure student readiness for college-level work and the workforce.
As competency-based and other student-centered reforms increasingly make their way into policy, empirical research lags behind.Research must keep pace with changes in policy and practice by providing evidence about both the implementation and the effects of competency-based reforms.This should include research that builds upon the current study to investigate the link between student-reported exposure to competency-based education and student outcomes.

Figure 1 .
Figure 1.Conceptual framework for the study.Note: The conceptual framework hypothesizes three potential manifestations of competency-based education implementation, as represented by three latent factor models.In Model A, dimensions underlying hypothesized elements of competency-based education are modeled as a set of interrelated latent factors, although no underlying explanatory structure is imposed on the interrelationships In Model B, it is hypothesized that interrelationships among these dimensions are explained by a set of second-order factors representing the hypothesized elements of competency-based education.Model C imposes a structure by which interrelationships among the 10 individual dimensions are explained by an overall competency-based education second-order factor.In the figure, shaded ellipses depict first-order latent factors, and unshaded ellipses depict second-order latent factors.Each first-order latent factor is measured via observed indicators (seeTable 1 and Appendix Table A1), which are not depicted.

Table 1
Standardized Loadings on Latent Factors Reflecting the Constructs of Interest and Model Fit Statistics

Table 1 (
Cont'd.) Standardized Loadings on Latent Factors Reflecting the Constructs of Interest and Model Fit Statistics Note: Source is the Competency-Based Learning Survey for Students, n=599.Bolded font indicates a modification made to Model A (original), resulting in Model A (revised).

Table 2
Note: Source is the Competency-Based Learning Survey for Students, n=599.All correlations significant at p≤.001.

Table 3
Source is the Competency-Based Learning Survey for Students, n=599.Item 1 wording is "Students in my courses all work on the same assignment at the same time" and Item 2 wording is "My teachers spend most of class time giving a lecture Note:

Table A1 .
Sample Descriptive Statistics on Survey Items, Organized by Original Constructs (Constructs Measured by Latent Factors)

Table A1 (
Cont'd.) Sample Descriptive Statistics on Survey Items, Organized by Original Constructs (Constructs Measured by Latent Factors)Item Construct/Item description M(SD) Personalization: Individual (PERS INSTR), as indicated by how often each of the following occurs in the student's current courses a : Flexible assessment (FLEX), as indicated by the number of times each of the following occurred in the students current courses c :

Table A1 (
Cont'd.) Sample Descriptive Statistics on Survey Items, Organized by Original Constructs (Constructs Measured by Latent Factors)Item Construct/Item description M(SD) Skills and dispositions: Perseverance (PRSVR), as indicated by how often each of the following occurs in the student's current courses a : Source is the Competency-Based Learning Survey for Students, n=599.a Response options included 0=Never, 1=Seldom, 2=Sometimes, 3=Often, 4=Always.b Response options included 0=None, 1=Some, 3=All or most.c Response options included 0=Not at all, 1=1-2 times, 2=3-4 times, 3=5 or more times. Note: