A National Study of School Effectiveness for Language Minority Students' Long-Term Academic Achievement Final Report: Project 1.1

Appendix B

Download PDF

Thomas-Collier Collaborative Research with School Districts: Student Program Participation
Data Collection Form for Research Stages 3-5

Wayne P. Thomas and Virginia P. Collier, George Mason University

Research funded by the Center for Research in Education, Diversity, and Excellence, U.S. Department of Education, Office of Educational Research and Improvement (OERI) from 1996-2001

The purpose of this data collection form is to capture program participation information in your school district. Pages 1-6 of this document explain the data collection process. The forms for data collection follow and may be duplicated as needed.

The Thomas and Collier research with collaborating school districts proceeds through five stages over a period of several years. These stages are completely explained in our "Five Stages" document which we have previously shared with you. In each stage, we address different research questions of interest and local school districts can insert their own local questions in each stage as well. Each stage also has its own information requirements, in the form of different types and forms of data needed to address each stage's research questions.

A quick review of these stages follows below in Table 1. For each stage of the sequential Thomas and Collier research model, Table 1 indicates the major intent of that stage, the primary research questions addressed in that stage, and the data required from school districts to address those research questions.

In Table 1, we use several acronyms to describe various categories of students who receive instructional services in each school district. These acronyms are based on the terms adopted by the federal government and may differ from those used in some states and districts.

Stage 1 of our collaborative research with your school district is a needs assessment that investigates how the three mutually exclusive groups have fared instructionally during the past five years in the school district's instructional programs. Stage 2 then looks only at LEP students and achievement gap closure, by age and by date of entry into the school district. No program comparisons are attempted in Stages 1 and 2, but program comparisons are the major focus in Stages 3-5.

Table 1

Stage 3 Data Collection

We have been planning or engaged in data collection for Stages 1 and 2 in your district in the past. Data collection for stages 1 and 2 has primarily focused on student characteristics and test scores over the past 5-10 years, in order to allow us to address Stage 1 and Stage 2 research questions. Since our research is long-term, requiring several years of data collection, we must collect and store data each school year on the program experiences that each student receives in order to address the Stage 3 questions that compare achievement trends and effect sizes for various program alternatives. Using this annual ëprogram participation' data, we can link student program experiences to long-term achievement test data that is collected during, and at the end of, the project. A greatly simplified form of this process might appear as follows:

Year 1 -- Collect student participation data (type of program experiences received by each LEP student) for Year 1 programs
Year 2 -- Collect student participation data for Year 2 programs
Year 3 -- Collect student participation data for Year 3 programs
Year 4 -- Collect student participation and test score data for each student for Year 4
Year 5 -- Calculate average test scores for students who participated in each program type in Years 1-4

It is now time to begin collecting Stage 3 data, information on which program types were received by each LEP student, by the end of each school year in June. This information should be captured now (or soon), in each school. When approaching the end of the school year, it is important to collect this information before the district's centralized student information system is "reset"during the summer prior to the beginning of the new school year in September. For most school districts, a professional who regularly observes in classrooms (e.g., a resource teacher) can provide a reality-based brief description of the typical instructional experience that will suffice for initial Stage 3 work. With each passing year, however, we plan to collect "finer-grained" data on the exact nature of instructional strategies utilized with language minority students, probably through the use of a teacher survey and/or teacher interviews.

Our overall objective in Stage 3 is to be able to document the types of instructional experiences received by each LEP student during each school year. Operationally, this means that we must be able to link each LEP student ID number with a description of a program type that was employed in the student's school (for school-wide LEP programs) or in the student's class (for situations where different program types are employed within the same school) for each school year under study. For example:

Case #1: for schools in which the same LEP program and instructional strategies are used school-wide, a download of the school's student ID numbers of the school's LEP students can be mass-matched with a code describing the program type (e.g., ESL taught through content) that is in use throughout that school. However, if there are substantial variations in how a program type is implemented and delivered from one classroom to another, the nature of these variations should be documented. For example, the program type called "ESL taught through content" can be delivered with substantial variations in teacher instructional strategies. In this case, the label "ESL taught through content" should be supplemented with class-by-class descriptions of how this program has been delivered to English language learners by individual teachers.

Case #2: for schools in which different LEP program and instructional strategies are used in classrooms within a school, we will need to link a teacher name or number to each student ID. To each teacher name or number, we will need to link a brief description of the LEP instructional practices utilized by each teacher.

How should program types be described?

In prior research and writing, we have referred to three major categories of program types. These include Enrichment Bilingual Programs, Remedial Bilingual Programs, and Remedial English-only Programs. To these, we can add Enhanced English-only Programs such as the Cognitive Academic Language Learning Approach (CALLA). Program types within each of these major categories may vary their names or labels by state and by district. The commonly encountered program types that we include in each of the four major program categories are listed below.

Category 1: Remedial Bilingual Programs - includes Transitional Bilingual Education, both Early-Exit (generally 2-3 years in length) and Late-Exit (generally 3-5 years in length)

Category 2: Remedial English-only Programs - includes ESL Pullout and ESL taught through content in elementary schools, and Sheltered ESL instruction in high schools, taught as a remedial subject for 1-3 periods per day

Category 3: Enrichment Bilingual Programs - includes one-way developmental bilingual education, two-way developmental bilingual education. Other names are dual-language programs, dual-immersion programs, bilingual immersion programs. Focus is on grade-level academic work across the curriculum, taught through the two languages, using interactive, discovery, hands-on learning.

Category 4: Enhanced English-only Programs - ESL taught through content or Sheltered ESL instruction (see Category 2 above) that also provides additional emphasis on student cognitive development; grade-level academic work across the curriculum; interactive, discovery, hands-on learning, or other instructional improvements to "basic ESL." Often this is done in a self-contained classroom for 1-2 years, or occasionally an ESL teacher teams with a mainstream teacher (both as equal partners in the teaching).

While program labels such as "ESL Pullout" are frequently employed, we and other researchers have found that there can be substantial variation within program categories when programs are actually delivered in classrooms to English language learners. In our prior work, we have attempted to draw attention away from program labels and instead to emphasize the features and characteristics of classroom instruction under each program type.

From our analyses, it appears that we should be able to characterize at least five factors in order to achieve a fully-specified LEP program description. These factors are presented in the form of questions (and possible answers) below. While instruction in any classroom can include a highly complex set of teacher-student and student-student interactions, we have found that useful distinctions among LEP program types can be framed by describing the classroom instruction in terms of answers to the following questions:

Factor #1: Is the teacher fully bilingual and capable of teaching in both English and in another language (e.g., Spanish)? Is the teacher certified in bilingual instruction? In ESL instruction? As a mainstream teacher?

Factor #2: Does the teacher teach English as a subject only; or does the teacher teach some subjects (e.g. math, language arts); or does the teacher teach all subjects (math, science, social studies, language arts)?

Factor #3: Does the student's teacher use the student's first language (other than English) in instruction? If so, approximately how much of the student's instructional time is in the student's first language?

If yes, the program is bilingual (TBE, one or two-way DBE)
If no, the program is English-only (ESL pullout, ESL taught through content, sheltered English instruction)

Factor #4: Does the teacher use cooperative learning strategies, both whole language and phonics-based reading instruction, variably-sized small-group instruction, non-textbook-based instructional materials, or other "current" instructional strategies? Or does the teacher use "traditional" homogeneous instructional strategies that rely primary on text-driven instruction in large groups? Does the teacher explicitly teach problem-solving skills in order to improve student cognitive development?

Factor #5: Are mainstream native-English speakers present in the LEP classroom (e.g., two-way DBE) or not?

As you work with this data collection form, you may add other variables and information that you consider important to collect, to help us define meaningful distinctions in program variations. We encourage your suggestions for improvement of the quality of data collected.

Thomas-Collier Collaborative Research with School Districts: Student Program Participation Data Collection Form

IMPORTANT CONTEXTUAL INFORMATION ABOUT YOUR SCHOOL OR STUDENTS that we should consider in our joint research with your school district:

Teacher Information

Teacher Name: ____________________Teacher Number: (if used) _______________

Teacher Credentials	Please answer Yes(Y) or No(N)
Teacher is certified to teach in mainstream classes.
Teacher is certified to teach bilingual classes.
Teacher is certified for ESL classes.
Teacher can teach only in English.
Teacher can teach only in another language. (which language?) ______________________
Teacher can teach in English and another language.
Teacher is strongly proficient in both English and another language.
Teacher is teaching using both languages in instruction this year.

ADDITIONAL INFORMATION ABOUT TEACHER CREDENTIALS that should be considered:

Both we and your school district thank you for your assistance! Please feel free to add any additional comments, questions, or suggestions in the space below. We would especially be interested in any questions (and your answer) that you believe that we should have asked, but didn't.

Back to Main Report

©2002 Center for Research on Education, Diversity & Excellence. All rights reserved.