Data and Performativity in Doctoral Education: Information Gaps and Suggestions for Overcoming Them

Since the mid-1990s, there has been an abrupt growth of doctoral enrollments and doctoral programs in Argentina and Latin America. However, completion rates at this education level are far from satisfying. Attrition rates in Social Sciences doctoral programs “are known” to be high, and higher in Social Sciences and Humanities (which will be later called “Soft Sciences”) than in STEM (which will be later called “Hard Sciences”) disciplines, although there are no valid and reliable data. In order to address this problem, we carried out an investigation with two objectives: (1) to measure performance (efficiency 1 This article is a revised version of a paper presented to the XII Argentine Meeting on Population Studies held at the Salta National University (Salta, Argentina) September 16-18, 2015. This is an unofficial translation and provided for reference only. Education Policy Analysis Archives Vol. 24 No. 124 2 or productivity) in 18 doctoral programs in STEM disciplines, the Social Sciences, and Humanities at various universities in Buenos Aires city and the surrounding Metropolitan Area; and (2) to design a procedure for collecting valid and reliable information that would allow us to assess doctoral programs' performance. This measure was defined along two dimensions: completion rates (regular time rates and extra time rates) and average time-todegree by cohort. We compiled data on individuals' academic tracks in each cohort from their enrollment to their completion or dropout by using paper-files and digitized databases provided by academic departments. Information was analyzed by program, by cohort, and by six grouped cohorts for 2001-2006. Substantive and methodological results were found. Among the substantive results, we discovered better performance in STEM disciplines than in the Social Sciences and Humanities. As to the methodological results, postgraduate statistical registering matrix was designed and a series of recommendations are presented for use by doctoral programs and graduate programs generally.


Introduction
Since the mid-1990s, Argentina experienced an abrupt growth in the supply of doctoral programs and in enrollment demand.Since the 1970s, the "knowledge society" has become relevant, as the production and circulation of knowledge has acquired prominence in a context where knowledge and the economy became closely linked (Castells, 1997).In Argentina, the Higher Education Law, passed in 1995, validated an educational model that emphasized the production and transmission of knowledge, as well the exponential growth of graduate programs.Apparently (if the statistics are trusted, although below we will question their reliability), the total supply of graduate courses more than tripled between 1994 and 2014, rising from 793 to 2098 (Fliguer & Davila, 2010;SPU, 2015).This movement was differential: graduate Specializations grew the most (from 301 to 964), followed by Masters' programs (from 246 to 742), whereas growth at the Doctoral level, was the lowest (246 to 392).The corresponding three types of graduates followed similar patterns with an abrupt rise between 1994 and 2002.Subsequently, these increases slowed, but the tendency was always increasing.
Similar growth in graduate programs took place in different world contexts.For example, Bowen & Rudenstine (1992), Ehrenberg et al. (2007Ehrenberg et al. ( , 2010) ) and Lovitts (2005Lovitts ( , 2008Lovitts ( ,2011) ) and Kiley note this phenomenon in the United States; Jiranek (2010), Halse and Mowbray (2009) observe it in Australia; and De Miguel, Sarabia Heydrich & Amirah (2004) also find similar trends in several European countries.For Latin American, Tuñón (2012) cites studies tracking the same trends in Venezuela, Colombia Mexico, and Chile.Moreover, they all agree that a large percentage of doctoral students fail to finish their programs, suffer frequent interruptions in their studies, and that the track duration is very long for those students who do graduate.
Growth in the supply of graduate programs has been accompanied by strong pressure from government agencies for accountability and performativity.In Argentina this trend is represented by the National Commission for Higher Education Evaluation and Accreditation (CONEAU, Comisión Nacional de Evaluación y Acreditación Universitaria) and the National Ministry of Science, Technology & Productive Innovation (MinCyT, Ministerio de Ciencia, Tecnología e Innovación Productiva de la Nación).However, doctoral completion rates are far from being satisfactory in Argentina and around the world.Furthermore, these rates differ by discipline: they are higher in STEM disciplines than in the Social Sciences and Humanities.
In terms of completion rates, Zainal Abiddin & Ismail (2011) cite Elgar's study about Canada.In Canada only 45% of doctoral students in the Arts and Humanities complete their program, while in Life Sciences and the Sciences (understood as STEM disciplines) the percentage of completion rises to 70% and 60%, respectively (2011).Wright and Cochrane present a similar scenario for the United Kingdom, where 51% of doctoral students in the Arts and Humanities complete their program, while in the Sciences (also understood as STEM disciplines) 64% of doctoral students reach this objective (2000cited in Zainal Abiddin & Ismail, 2011).In the United States, graduation rates vary between 30% and 50% in different fields within the Humanities; between 50% and 70% in STEM disciplines, (Lovitts, 2001); and between 35% and 60% in the Social Sciences (Gardner, 2013).Gardner cites Lovitts (2001), Golde (2005Golde ( , 2007) ) and Millett & Nettle (2006) to note that during the 1990s completion rates in the US varied between 11% and 68% depending on the field.For Australia, Kiley (2011) observed that only 50% of the doctoral students who began their studies between 2005 and 2011 had completed their programs.Jiranek (2010), who also studied Australia, found completion rates in the Natural Sciences to stand between 60% and 70%, and from 49% to 55% in the Social and Human Sciences.De Miguel, Sarabia Heydrich & Amirah (2004) found a similar scenario in Spain: in the 1990s, Experimental, Social and Human Sciences showed completion rates of 52%, 23% and 14% respectively.As this brief review shows, figures vary across disciplines and across countries, but there are two common trends: completion rates do not reach 80% and they are consistently higher in Natural Sciences (the STEM disciplines) than in Social Sciences and Humanities.
We must stress that the availability of complete, valid and reliable information at the level of higher education is very low.In this context "it is known" that attrition is high and that attrition rates are higher in the Social Sciences and Humanities than in the Natural Sciences.In addition, "it is known" that attrition rates are higher at the dissertation-production stage than at the stage of coursework (giving rise to the "All But Dissertation"/ABD concept).These claims are based on abundant observations in few cases, which are collected with little or no systematic methodology.
The absence of valid and reliable statistics in part is due to characteristics of the graduate educational level.Unlike the primary and secondary levels, higher education at the graduate level is not obligatory, has no strict terms of completion, abounds in extensions and leaves, and allows "unregistered migrations" between programs within and/or between academic units.For example, often, there is no single starting date that makes it possible to identify cohorts and calculate the stock of students at any given moment.The absence of valid and reliable statistics also reflects the great heterogeneity within postgraduate programs, particularly in doctoral programs: these programs may have a structured, semi-structured, or customized format; they may or may not have a cohort regime (same date initiation); and dissertation work may be simultaneous to coursework or subsequent to completion of coursework, etc.Furthermore, many academic departments place little importance upon statistical information as a necessary tool for tracking and monitoring postgraduate programs and for policy evaluation.On the other hand, accrediting agencies value statistical information, as they emphasize accountability.In Argentina, the CONEAU is interested in this evaluation, something which seems to be common throughout the world, at least for the countries for which we have information.
In Argentina, as we have already noted, since the 1990s there has been a strong focus on higher education evaluation following the guidelines of international agencies (Krostch, 2002).As a result major changes were introduced in two dimensions in universities: on the one hand, in general funding, teaching and research and, on the other, in increasing accountability (Escotet & Aiello, 2010;Fernández Lamarra, 2003).
As regards the first dimension, some of the policies put into practice were: the creation of the Higher Education Quality Improvement Fund (FOMEC, Fondo para el Mejoramiento de la Calidad Universitaria) ; and the Incentives Program for Research-Professors ; an increase in the number of doctoral scholarships offered by the National Council for Scientific and Technical Investigations (CONICET, Consejo Nacional de Investigaciones Científicas y Técnicas) and the National Agency for the Scientific and Technical Promotion (ANPCyT, Agencia Nacional para la Promoción de la Ciencia y la Tecnología) .
As for the second dimension, the creation in 1993 of the Higher Education Policy Secretariat (SPU, Secretaría de Politicas Universitarias), within the Ministry of National Education, deserves special attention.The policy cluster promoted under the wing of the SPU constitutes the new system of higher education.This system pursues a strong emphasis on the development and evaluation of graduate programs' quality and is associated with the creation of a Higher Education Information System (SIU, Sistema de Información Universitaria) to optimize performance and efficiency.In response, the SPU should systematize statistical information provided by universities and publish yearbooks or graduate guide-books for researchers and civil society.
The SPU's tasks are combined with the assessment and accreditation system established by the CONEAU in 1995 to evaluate operating and proposed universities, and to regulated undergraduate and graduate programs through their accreditation.This secretariat on higher education pursues three objectives: "to promote the consolidation and evaluation of the graduate system following internationally recognized criteria of excellence; to promote the training of highly qualified human resources, for both academic teaching and research activities, as well as professional specialization; and to offer society reliable information about graduate educational quality, so to expand the capacity of prospective student's to make choices" (Marquis, 2009, 50).The CONEAU, which is regulated by the Higher Education Law (LES, Ley de Educación Superior), is structured to give the state a major role in evaluating higher education and has evaluation as a central criteria for its policy (Krotsch, 2002).
We maintain, as did Marquis (2009, 39), that "there is little consistent and reliable information about postgraduate programs for Argentina" despite this cluster of policies.This comment echoes an observation made for another context: "UNESCO, OECD, and even EUROSTAT´s global statistics do not give reliable data on the doctoral level [...]There are therefore no comparable international statistics."(De Miguel et al. 2004, p.148).
The available data does not allow us, among other things, to measure retention and student completion in both undergraduate and graduate programs, nor to pursue systematic studies about causes of attrition.

An Attempt at Valid Measurement of Doctoral Program Efficiency
We sought to measure doctoral program efficiency in various disciplines in terms of completion rates and average time-to-degree by cohort using data that we collected.To contextualize, we note that this work was part of a research program aimed at producing knowledge about how researchers are trained.This research included the following aspects: institutional and pedagogical factors, "learning environments", thesis strategies for students, and training provided by dissertation supervisors and theses workshops/seminar teachers.We focused on doctoral programs because although possessing a doctorate is not necessarily a requirement for being a researcher, generally having a doctoral degree is the entry requirement for passage into the academic research community.
In other words, we carried out research with two objectives: (1) to evaluate the performance (efficiency or productivity) of 18 doctoral programs in the Social Sciences and Humanities and in the Natural Sciences from in different academic departments from five universities (these universities, located in Buenos Aires and its Metropolitan Area, included public and private institutions); and (2) to design a methodology to produce valid and reliable information on the performance (efficiency or productivity) of graduate programs to record data for their monitoring.
We defined program performance in terms of two dimensions: (i) completion and (ii) timeto-degree.We measured completion (i) with two indicators: (i.a) overall completion rates (OCR) and (i.b) regular (set by each doctoral program) time completion rates (RTCR).The second dimension, time-to-degree (ii), was measured in terms of average time-to-degree (TTD).Since programs are differently organized, we defined a cohort as the group of students who are admitted in a calendar year regardless students´ different admission dates within the same year.In consequence, this enabled calculations for each cohort.
After examining the quality of the information produced by the SPU and its data loading system (SIU, Sistema de Información Universitaria), we decided to design a methodology to measure efficiency.Universities are required to collect data from undergraduate and postgraduate programs for SIU and send them to the SPU to be processed and published in statistical Yearbooks.Particularly, a first Yearbook exclusively on graduate programs (masters and doctoral degrees) was published in 1997 within the Higher Education's Information System Improvement Program (PMSIU, Programa de Mejoramiento del Sistema de Información Universitaria).Although PMSIU ran between 1997 and 2004, it did not produce information until 2006.After eight years of inactivity, statistics were published annually from 2006 to 2012.
In addition to the interruptions which do not provide statistical series, there are no mechanisms that oblige universities to send information to the SPU nor to rigorously load data into the SIU.As a consequence, certain yearbooks clarified that certain universities had been excluded from the publication for diverse reasons, such as: (i) failure to regularly collect this information at the graduate level; (ii) failure to accredit their graduate offerings; and (iii) failure to respond to SPU's requests for information.Both in Estadísticas 1997 (Statistics, 1997) and in Anuario 2006 de Estadísticas Universitarias (Higher Education 2006 Yearbook), it is noted that "data collection and its consolidation meant an effort for universities because [...] there is no precise, in quantitative terms, knowledge about the postgraduate level, since such information is not traditionally collected by statistics departments in institutions of higher education."The Anuario 2008 de Estadísticas Universitarias (Higher Education Statistics 2008 Yearbook) noted that "[publishers] cannot present historical series due to the lack of information that might rigorously illustrate postgraduate enrollment growth, in recent years.[They] hope to achieve greater coverage and thus be able to have reliable historical information in time."(SPU, 2008, 125).Similarly, the Anuario 2010 de Estadísticas Universitarias (SPU, 2010) clarifies that "even though measuring the students population [referring to graduate students] has been improving data coverage, there are institutions that still supply partial information" (p. 7).
Other authors have been already warned about these points.Marquis (2009) sustained that "[SPU] information was built on data provided by universities that did not necessarily use the same criteria and procedures to collect their data, and there were time periods for which they did not report information or they did so in an incomplete way" (p.39).García de Fanelli (2004) and Kisilevsky (1999) noted the asymmetry in information between universities and the government: the universities have more information than the government agencies.Regarding this problem, the SPU itself in several Yearbooks has warned that "the information in this chapter may be under representative".The 2006 Yearbook said that: "today, 39% of the national universities that offer postgraduate degrees have not reported the number of students for at least one year during the period 1997-2005, during which there is no information for periods of three, four, and more years".This lack of data makes it impossible to build a valid view of reality; however, the SPU considers that its reports provide an interesting database, even with gaps that require improvement.
Many researchers have used information drawn from these yearbooks to analyze the evolution of graduate programs without warning readers about such shortcomings.Thus, several works using the same SPU statistics agree on the increase in supply of graduate education, but they do not agree on the absolute numbers.On this subject, Tuñón (2012) referred to Barsky's work (1999) and noted that "there are data in the same study that show a difference in the total supply of almost 100 postgraduate programs in absolute numbers, representing a 5% variation in percentage terms" (p.30).
In addition to the problems mentioned above, the SPU does not produce adequate information about the efficiency of the system.The doctoral completion/attrition rates that are calculated and published are not really indicators of completion and attrition since they represent the relationship between the number of new enrolled students and the number of completing-students in a calendar year.In other words, it compares the cumulative stock of students (enrolling students) with accumulated completing-students stock.This implies that "efficiency" is not measured in terms of the ratio between input and output.To measure the latter, we must know the number of students who graduated after a period of time out of those who were enrolled in the same year; i.e. the number of completing-students of the same cohort.
In sum, the SPU occasionally provides information, of questionable validity and reliability, with faulty coverage, that is not comparable over time.This situation explains why we were forced to produce our own data and, by doing so, design specific processing techniques.
As soon as we started our research, it became clear that we were going to be unable to analyze the system as a whole because the population of Ph.D. students (and the population of students in higher education in general) is an 'open' population in which (unlike the primary and secondary levels) permanence within the Argentina's graduate system is not recorded.For example, if a student does not complete a program, it is impossible to know if he/she migrated to another program or if he/she dropped out of the system (this is possible for the primary or secondary level in since "migration" between schools is recorded within this country).This means that what we evaluated was each doctoral program's efficiency; we did not study completion or attrition conditions of each individual that went through a doctoral program.Global doctoral efficiency, understood as completion of studies (which is accurate) and not attrition (which is inaccurate), can only be assessed by cumulating information from all program.
The construction of such statistics was a complex task, especially for doctoral programs, due to the lack of records registering permanence, as well as for other reasons, such as the absence of shared definitions regarding curricular and organizational structure.Even though the regulations of almost every doctoral program define a time limit for completion, these time periods are not identical, not nor are there homogeneous policies regarding extensions, leaves, and readmissions.Nevertheless, data was available from the universities and we used it, but only after evaluating and cleaning it before processing.It should be stressed that we sought to evaluate the efficiency, not the quality of doctoral programs.

Methodological Aspects
We selected the 18 doctoral programs based on theoretical and pragmatic criteria.The first led us to include different disciplinary fields; the second led us to take into account the accessibility of the data.There is an abundant literature on the theoretical criteria.Some studies are stronger than others in terms of their empirical basis, but all studies agree that doctoral completion rates for the Natural Sciences are higher than those in the Social and Human Sciences.There are multiple reasons for this finding.Perhaps, "learning environments" and working "practice communities" are the main reasons, as well as differential access to scholarships among diverse disciplinary fields.If comparison in efficiency between fields revealed systematic differences, it would allow us to speculate about the factors involved in such events.Moreover, it would take us closer to our ultimate goal, which is to design strategies (educational, institutional, organizational, curricular) aimed at improving the performance of doctoral student training and to lower attrition.
We collected data for each individual student throughout his/her academic career (identified by a numeric code in replacement of first and last name) and then we added them by cohort or initiating year into the program in order to calculate completion rates and average TTD.Thus, the unit of measurement was the student and the unit of analysis was the cohort of students admitted in the same year.Hence, we designed a data collecting matrix using each doctoral students' academic tracks and milestones (see Figure 1).As it was said, our research was intended to measure not only efficiency, but also to identify milestones at which points program attrition was higher.Paying special attention to these points might have offered an enriching perspective to design efficiency improvement strategies.Unfortunately, we could only identify such milestones in only a few of the 18 doctoral programs because of a lack of original records.In consequence, these aspects were not included in this paper.In most programs investigated this approach led us to an input-output study; thus, a "black box" study, by considering the number of incoming (admission date) and the number of graduates (dissertation date).If intermediate milestones information had been accessible, it would had been possible to go into the "black box." As we said, we studied 18 doctoral programs at universities from the public and private sector located in the Buenos Aires's Metropolitan Area (AMBA, Área Metropolitana de Buenos Aires) in Soft Sciences -seven programs -and Hard Sciences -11 programs.We used Becher's classical works (1989,1993), Becher & Kogan (1992), Light (1974), Biglan (1973), Lodhal and Gordon (1972), Ladd and Lipset (1976) and Kolb (1981) to select diverse disciplinary fields.Becher himself took references from these works' models and elaborated a new classification, which is adopted in this research.This classification is organized in poles: hard-soft and pure-applied.However, we only took hard-soft pole.
Hard disciplines are sometimes described as cumulative: they pursue atomistic knowledge (tree-shaped or pyramid-shaped); generally, they care about universal topics, quantities and simplification, and specific matters; and results are associated with discoveries and explanation.In addition, Hard disciplines are moved by pragmatic purposes; they care about the domain of the physical environment; often, their results are products and techniques.Soft disciplines are repetitive and holistic; they care about specific topics and quality; and results are associated with comprehension and understanding.Moreover, Soft disciplines are also functional and utilitarian.They care about professional practices, and results are based on protocols and procedures (Becher, 1993).
In order to facilitate interdisciplinary comparison, we added six starting cohorts in each program between 2001 and 2006.The year 2001 was selected as a starting point on the basis of the creation date of the most recent program.Given that there are regulations that allow for up to eight years for completion (with the inclusion of two one-year extensions) we selected 2006 as an end point to allow cohorts from all 18 programs to achieve completion at the time that we collected data (2014).Programs varied in regards to management sectors, i.e. whether they were private or public; age as measured from year of creation; size in terms of enrollments; curricular structure, i.e. whether they involved custom designs, semi-structured, or structured formats; the modality of education, i.e. face-to-face, blended, or virtual; the nature of the dissertation production period --alongside coursework or subsequent to completion of courses; and the amount of time allotted for dissertation completion.These program characteristics are summarized in Table 1.
Before collecting the data, we acquainted ourselves with each program's regulations, and curricular and organizational structure.To do this, we analyzed documents and interviewed program directors or academic assistants.As shown in Table 1, all of the "hard" programs are run in public universities.Whereas, "soft" programs are found within public and private universities.Most programs are customized; all of them are face-to-face; most of them separate the stage of coursework from the stage of thesis preparation; and regulations for completion specify periods that vary between 4 and 8 years, including permissible time extensions.We produced three indicators for the completion dimension: overall completion rate (OCR), regular time completion rate (RTCR) and extra time completion rate (ETCR).The first is defined as the ratio between the number of graduates and the number of admitted students over a period of time; RTCR measures the number of graduates that completed the program in regular time; while ETCR is a measure of the number of graduates that completed the program after regular deadlines.Therefore, OCR is equal to the sum of RTCR and ETCR.
We developed only one indicator for the time-to-degree dimension: the global average timeto-degree.This is the average time taken by all graduates within a set who began at the same date (year) to complete their PhDs.(i.e. the difference between the 'dissertation date' and the 'admission date').
As regards these indicators, it is important to note that the population studied by these completion rates -OCR, RTCR and ETCR -and the average time-to-degree always encompasses the set of admitted students in a delimited period of time; in this case, all individuals admitted between 2001 and 2006.In other words, it is the temporal definition of admission that defines the population studied.This clarification has substantive implications because it differentiates these indicators from those used by the SPU.As mentioned before, this organization works with cumulated graduates stocks per year.As already stated, our results are based on a study of 18 doctoral programs: 11 from fields in the "Hard sciences" and seven from fields in the "Soft Sciences" (Becher, 1993) (see Table 2).

Disciplinary
OCRs from these 18 doctoral programs vary between 9% and 100%, with systematic differences between disciplines.OCR rates among the Hard sciences vary between 45% and 100%, while these rates for the Soft sciences vary between 9% and 100% (see Table 2).Within the first group, Physical science 2 has the highest OCR, while Computer science has the lowest (45%).In the second group, Social science 1 and 2 (55% and 57%, respectively) have the highest OCR and International Relations has the lowest (9%).
Note that if Computer science were excluded from the Hard Sciences group, the Soft Sciences program with the highest OCR, would have a rate lower than the program with the lowest OCR in the Hard Sciences.This initial reading of the data clearly points to differences by disciplinary fields in regards to completion rates within efficiency levels.
In the Hard or Natural sciences the TTD dimension reveals a pattern similar to that of the OCR: Physical science 2 has the lowest average time to degree (4.4 years) and Computer science has the highest (5.58 years).This pattern is not repeated in the Soft sciences: the program with the lowest TTD, Sociology (4.41 years), does not have the highest OCR (Social sciences 2).In addition, the lowest TTD program (Sociology, 4.41 years) is far from having the highest OCR (Social science 1 and 2, 55% and 57%, respectively).
It is worth noting that even though TTDs for the Hard sciences are lower than those of the Soft sciences, the differences between these times are smaller than the differences among OCRs.Perhaps, RTCRs combine indicators from both efficiency dimensions because they consider time deadlines established by academic departments.
We can detect two patterns of behavior for the Hard Sciences with these measures.First, RTCRs correlate identically with OCRs.Thus, ordering programs by highest-to-lowest OCR reiterates the program ordering by RTCR.Secondly, differences between both rates are very small, 5% points at most.This means not only that a greater proportion of students graduate in doctoral programs in the Hard sciences than in Soft science programs, but that students complete their degrees in most programs by respecting regular deadlines.Only a minimum fraction of students complete doctoral programs outside the time limits established by academic departments.The situation in the Soft sciences is very different.Differences between program OCRs and RTCRs are very broad.This difference implies that only a minority of students graduate by regular time deadlines.This pattern is reflected in another characteristic: the highest-to-lowest ordering of programs by OCR and by RTCR does not coincide.
The efficiency levels found here reproduce similar findings in other contexts such as the United States (Bowen & Rudenstine, 1992;Ehrenberg, Zuckerman, & Groen, 2010;Ferrer de Valero, 2001;Gardner, 2008Gardner, , 2009Gardner, , 2010;;Golde, 2005Golde, , 2007;;Kim & Otts, 2010;Main, 2014;Millett, 2006;Vassil & Solvak, 2012): doctoral programs in the Hard sciences (Natural Sciences in OECD terms) have higher efficiency levels than programs in the Soft sciences (Social sciences and Humanities in OECD terms).In addition, the behavior of the indicators is more homogeneous in than Hard sciences than in the Soft.
We hypothesize that differences in efficiency levels between disciplinary fields are related to several factors discussed in Becher's studies about academic tribes (Becher, 1989;1993;Becher & Kogan, 1992): initiation modes, forms of social interaction, the nature of knowledge in each discipline, and access to financing.So far we have presented an efficiency evaluation of doctoral programs, our first objective.As for the second objective, the design of procedures to carry out an efficiency evaluation, we list several recommendations that emerged from our experience and discuss challenges.
The first and foremost recommendation is to gain information access.Even though some academic departments have records and are willing to show them subject to individual data confidentiality restrictions, other departments do not maintain valid and complete records or they do not process them.Some departments are very reluctant to provide such data for research either because they are unaware of its importance, fear that doing so will disrupt their day-to-day operations or, probably, fear discovery and publicity regarding the low efficiency of their doctoral programs.
Second, attrition evaluation is only possible for each individual program, not for the doctoral system as a whole.This limitation is due to the absence of information with which to determine whether a doctoral student who does not complete a program is dropping out or is migrating to another doctoral program within the system.
Third, since doctoral programs do not always record when students drop out , to study attrition it is necessary to trace in each cohort each students' academic track from date of admission and then detect when the student disappears from the records.
Fourth, to develop a valid calculation of overall performance and performance at different stages, it is necessary to have information about each doctoral programs' regulatory, organizational, and curricular characteristics (structured, semi-structured or custom), dissertation preparation periods (after courses or simultaneously), academic track milestones, and the associated regular time deadlines, among others.
Fifth, there may have been changes in the course of a program's history, for example, in deadlines for completion or in the requirements at each intermediate curricular stage.Therefore, in such cases it may be necessary to correct the completion-attrition criteria used when studying program efficiency in historical series.
Sixth, after verifying the existence of data, its location, and the manner of its recording, the researcher should use the most primary and primary and disaggregated sources so as not to lose information.
Seventh, it is necessary to investigate how regulations are put into practice.It is very important to understand mechanisms used to "skip" program regulations; for example, when a person exceeds deadlines and fails to complete their dissertations, why isn't the person considered a drop out?
Eighth, we recommend distinguishing between two completion times: regular-time and overall completion time.When doctoral students take longer than they "should", the system is not failing at getting the student to complete the program, but the system is performing "less efficiently".
Ninth, counting months and counting years produce different results when calculating TTD and completion rates.In addition, if years are counted the results differ according to whether one counts the year that is beginning or counts only completed years.Counting months is a more accurate way to measure students' academic tracks.
Tenth, when there is a discrepancy between the date specified in a database and the date found in an official resolution for the same stage, we recommend using the oldest date (usually in the database) as resolutions are often delayed for years.
These are several methodological challenges that we faced in our study.Although there are many challenges, we stress the importance of measuring the efficiency of doctoral programs.Such measurements provide a powerful tool for evaluating performance.Institutions need to increase their awareness regarding the importance of measuring efficiency, in order to understand their own strengths and weaknesses, and to improve "their service" and performance.These tasks require that qualified personnel collect relevant data.Doctoral programs, regardless of their size, should record dates of: admission, registration, advisor appointment, dissertation supervisor appointment, and project proposal approval, completion of the dissertation, and dissertation defense.These records should also include information related to the student's coursework, including: lists of the courses in which the student enrolled, enrollment dates, approval dates, and the number of credits granted.Once such data is recorded, they must be processed for the bodies supervising the program, and they should be made available for public access in the interest of complying with desirable transparency requirements.

Discussion
We observed that completions rates (OCR, RTCR and ETCR) and time-to-degree for the Hard science disciplinary fields were better than those rates and times of the Soft sciences by studying 18 doctoral programs in Argentine universities.Observation of these differences do not lead us to rank one disciplinary field over another, nor to recommend transferring one field's working models to another, because generally disciplines are organized around very different and unavoidable epistemic differences.However, we are aware that the Soft sciences are characterized by higher attrition and by longer completion times.These problems deserve an improvement trial.
Although after months of lobbying we were welcomed in several academic departments, our experience revealed a series of problems with data production at the doctoral level: (1) in some departments we discovered an absence of valid and reliable information; (2) authorities were often unaware of the need to record this information for departmental management; and (3) as a consequence, such information never made it to the national agency responsible for receiving, processing, and publishing this information for society's knowledge.On the other hand, our experience revealed that with persistence, wit, and tolerance researchers can overcome institutional resistance and gain access to the information needed for evaluating the efficiency of doctoral programs.
We insist that studies like ours are needed to provide valid and accurate baseline data.We need such data before we can attempt to evaluate factors that might be associated with attrition, identify those teaching strategies that are conducive to completion, as well as to identify at which stages of the academic track curricular and pedagogical changes are needed to improve doctoral efficiency.
We agree with Moler (2008), the SPU's Research and Statistical Information coordinator, when she warned that "educational statistics are an essential tool to generate effective knowledge about the reality of higher education [...] since it is necessary to obtain a correct evaluation of the impact, progress, and setbacks within the national university system (p.263.)We also recognize that the "construction of information systems in all organizations requires profound cultural changes.If we do not acknowledge this point [...], we will not understand the situations that directly affecting the compilation of information" (Moler, 2008, p. 265).
Finally, we want to stress that we study efficiency levels, their determinants, and associated variables not only with the goal of quantifying the information.Rather we are also interested in studying the functioning of doctoral programs because we are committed to improvements that contribute to increasing the number of scientific researchers in Argentina in the context of a "knowledge society" and to assuring the efficient use of always scarce financial resources.
In addition, we sustain that once data about completion, attrition and duration are produced, these might enable the study of factors that hinder and facilitate degree completion.Furthermore, it would allow to design curricular or pedagogical mechanisms to overcome greater attrition stages in academic tracks in diverse fields.

Table 1
Doctoral program characteristics for 2001-2006 cohortsThere are two Physical Sciences programmes (1 & 2) and two Social Sciences programmes because they take place in different universities. Note:

David Carlson, David R. Garcia, Margarita Jimenez-Silva, Eugene Judson, Jeanne M. Powers, Iveta Silova, Maria Teresa Tatto
education policy analysis archives editorial board Lead Editor: Audrey Amrein-Beardsley (Arizona State University) Editor Consultor: Gustavo E. Fischman (Arizona State University) Associate Editors: (Arizona State University) Cristina Alfaro San Diego State