University of California, BerkeleyGSE Home



    
how to apply faculty news events
programs courses research administration resources

prospective students
alumni & visitors
current students
faculty & visitors
 

A National Study of School Effectiveness for Language Minority Students' Long-Term Academic Achievement Final Report: Project 1.1

Appendix B

Download PDF

Thomas-Collier Collaborative Research with School Districts: Student Program Participation
Data Collection Form for Research Stages 3-5

Wayne P. Thomas and Virginia P. Collier, George Mason University

Research funded by the Center for Research in Education, Diversity, and Excellence, U.S. Department of Education, Office of Educational Research and Improvement (OERI) from 1996-2001

The purpose of this data collection form is to capture program participation information in your school district. Pages 1-6 of this document explain the data collection process. The forms for data collection follow and may be duplicated as needed.

The Thomas and Collier research with collaborating school districts proceeds through five stages over a period of several years. These stages are completely explained in our "Five Stages" document which we have previously shared with you. In each stage, we address different research questions of interest and local school districts can insert their own local questions in each stage as well. Each stage also has its own information requirements, in the form of different types and forms of data needed to address each stage's research questions.

A quick review of these stages follows below in Table 1. For each stage of the sequential Thomas and Collier research model, Table 1 indicates the major intent of that stage, the primary research questions addressed in that stage, and the data required from school districts to address those research questions.

In Table 1, we use several acronyms to describe various categories of students who receive instructional services in each school district. These acronyms are based on the terms adopted by the federal government and may differ from those used in some states and districts.

TermExplanation
LEPThe acronym ‘LEP' refers to those students who have been classified by the local district as "limited English proficient" (English Language Learners) and who have participated in district instructional programs designed to meet their needs.
ELLEnglish Language Learners -- generally synonymous with LEP
LMThe acronym ‘LM' refers to Language Minority students (those who speak a language other than English at home).
  'LM-but-not-LEP'refers to students who qualify as Language Minority but who have not been classified as LEP by the school district.
NESThe acronym ‘NES' refers to students who are native speakers of English and who are not LEP (limited English proficient) and not LM (language minority).

Stage 1 of our collaborative research with your school district is a needs assessment that investigates how the three mutually exclusive groups have fared instructionally during the past five years in the school district's instructional programs. Stage 2 then looks only at LEP students and achievement gap closure, by age and by date of entry into the school district. No program comparisons are attempted in Stages 1 and 2, but program comparisons are the major focus in Stages 3-5.

Table 1

Overview of Five-Stage Thomas and Collier Research Model
in Participating School Districts

Stage

Major Intent

Primary Research Questions

Data Needed

One

to document the district's past achievement outcomes for three mutually exclusive groups of students and to compare the five-year progress of the three groups. This is also known as the Thomas-Collier Test of Equal Educational Opportunity:

(1) former LEPs (English learners)

(2) students who are Language Minority (LM) but not LEP (did not participate in a local LEP program)

(3) native English speakers who are not part of groups (1) or (2) above

After five years of appropriate instruction in the district, is there an achievement gap between former LEPs (English language learners) and native-English speakers?

Has the achievement gap between former LEPs, LM-but-not-LEPs, and native-English speakers widened, narrowed, or remained the same for the past 5 years?

downloads of test scores and student classification information from prior years

Specifically:

(1) student ID

(2) original student classification

(3) date entered school

(4) test scores from recent years

       

Two

to document the past and present achievement performance of LEP students (English Learners)

Do LEP students close the achievement gap with each passing year?

Do older LEP students close the achievement gap differently from younger students?

Additional student information

Specifically:

(1) date of birth

(2) days attended school each year

       

Stage

Major Intent

Primary Research Questions

Data Needed

Three

to determine the average annual long-term achievement gains of former LEP students who participated in various types of programs for LEP students

Which programs allow students to close the achievement gap over time and which do not?

Do students in some programs close the achievement gap better or faster than in other programs?

What is the average sustained gain per year for each program?

Student program participation data

Specifically:

(1) program type(s) student received each year

       

Four

to enhance external validity (generalizability) and robustness of conclusions from stages One through Three by adding longitudinal cohorts, using cross-validation strategies, and employing resampling strategies.

Are the observed between-group and between-program differences in student achievement trends stable and consistent across comparable but different longitudinal cohorts of students during the past 5-10 years?

Stage 1-3 data for additional student cohorts

       

Five

to determine the long-term achievement of LEP students who received selected LEP programs in the past with control of pertinent extraneous variables on the enhanced data sets from Stage Four

With selected extraneous variables controlled using sample selection, blocking, or ANCOVA (if appropriate), are there long-term differences in student achievement among programs?

Student characteristics and other variables to be controlled

Specifically:

(1) initial grade placement in school

(2) free-reduced lunch for each year

(3) initial test scores at beginning of schooling

Stage 3 Data Collection

We have been planning or engaged in data collection for Stages 1 and 2 in your district in the past. Data collection for stages 1 and 2 has primarily focused on student characteristics and test scores over the past 5-10 years, in order to allow us to address Stage 1 and Stage 2 research questions. Since our research is long-term, requiring several years of data collection, we must collect and store data each school year on the program experiences that each student receives in order to address the Stage 3 questions that compare achievement trends and effect sizes for various program alternatives. Using this annual ‘program participation' data, we can link student program experiences to long-term achievement test data that is collected during, and at the end of, the project. A greatly simplified form of this process might appear as follows:

Year 1 -- Collect student participation data (type of program experiences received by each LEP student) for Year 1 programs
Year 2 -- Collect student participation data for Year 2 programs
Year 3 -- Collect student participation data for Year 3 programs
Year 4 -- Collect student participation and test score data for each student for Year 4
Year 5 -- Calculate average test scores for students who participated in each program type in Years 1-4

It is now time to begin collecting Stage 3 data, information on which program types were received by each LEP student, by the end of each school year in June. This information should be captured now (or soon), in each school. When approaching the end of the school year, it is important to collect this information before the district's centralized student information system is "reset"during the summer prior to the beginning of the new school year in September. For most school districts, a professional who regularly observes in classrooms (e.g., a resource teacher) can provide a reality-based brief description of the typical instructional experience that will suffice for initial Stage 3 work. With each passing year, however, we plan to collect "finer-grained" data on the exact nature of instructional strategies utilized with language minority students, probably through the use of a teacher survey and/or teacher interviews.

Our overall objective in Stage 3 is to be able to document the types of instructional experiences received by each LEP student during each school year. Operationally, this means that we must be able to link each LEP student ID number with a description of a program type that was employed in the student's school (for school-wide LEP programs) or in the student's class (for situations where different program types are employed within the same school) for each school year under study. For example:

Case #1: for schools in which the same LEP program and instructional strategies are used school-wide, a download of the school's student ID numbers of the school's LEP students can be mass-matched with a code describing the program type (e.g., ESL taught through content) that is in use throughout that school. However, if there are substantial variations in how a program type is implemented and delivered from one classroom to another, the nature of these variations should be documented. For example, the program type called "ESL taught through content" can be delivered with substantial variations in teacher instructional strategies. In this case, the label "ESL taught through content" should be supplemented with class-by-class descriptions of how this program has been delivered to English language learners by individual teachers.

Case #2: for schools in which different LEP program and instructional strategies are used in classrooms within a school, we will need to link a teacher name or number to each student ID. To each teacher name or number, we will need to link a brief description of the LEP instructional practices utilized by each teacher.

How should program types be described?

In prior research and writing, we have referred to three major categories of program types. These include Enrichment Bilingual Programs, Remedial Bilingual Programs, and Remedial English-only Programs. To these, we can add Enhanced English-only Programs such as the Cognitive Academic Language Learning Approach (CALLA). Program types within each of these major categories may vary their names or labels by state and by district. The commonly encountered program types that we include in each of the four major program categories are listed below.

Category 1: Remedial Bilingual Programs - includes Transitional Bilingual Education, both Early-Exit (generally 2-3 years in length) and Late-Exit (generally 3-5 years in length)

Category 2: Remedial English-only Programs - includes ESL Pullout and ESL taught through content in elementary schools, and Sheltered ESL instruction in high schools, taught as a remedial subject for 1-3 periods per day

Category 3: Enrichment Bilingual Programs - includes one-way developmental bilingual education, two-way developmental bilingual education. Other names are dual-language programs, dual-immersion programs, bilingual immersion programs. Focus is on grade-level academic work across the curriculum, taught through the two languages, using interactive, discovery, hands-on learning.

Category 4: Enhanced English-only Programs - ESL taught through content or Sheltered ESL instruction (see Category 2 above) that also provides additional emphasis on student cognitive development; grade-level academic work across the curriculum; interactive, discovery, hands-on learning, or other instructional improvements to "basic ESL." Often this is done in a self-contained classroom for 1-2 years, or occasionally an ESL teacher teams with a mainstream teacher (both as equal partners in the teaching).

While program labels such as "ESL Pullout" are frequently employed, we and other researchers have found that there can be substantial variation within program categories when programs are actually delivered in classrooms to English language learners. In our prior work, we have attempted to draw attention away from program labels and instead to emphasize the features and characteristics of classroom instruction under each program type.

From our analyses, it appears that we should be able to characterize at least five factors in order to achieve a fully-specified LEP program description. These factors are presented in the form of questions (and possible answers) below. While instruction in any classroom can include a highly complex set of teacher-student and student-student interactions, we have found that useful distinctions among LEP program types can be framed by describing the classroom instruction in terms of answers to the following questions:

Factor #1: Is the teacher fully bilingual and capable of teaching in both English and in another language (e.g., Spanish)? Is the teacher certified in bilingual instruction? In ESL instruction? As a mainstream teacher?

Factor #2: Does the teacher teach English as a subject only; or does the teacher teach some subjects (e.g. math, language arts); or does the teacher teach all subjects (math, science, social studies, language arts)?

Factor #3: Does the student's teacher use the student's first language (other than English) in instruction? If so, approximately how much of the student's instructional time is in the student's first language?

If yes, the program is bilingual (TBE, one or two-way DBE)
If no, the program is English-only (ESL pullout, ESL taught through content, sheltered English instruction)

Factor #4: Does the teacher use cooperative learning strategies, both whole language and phonics-based reading instruction, variably-sized small-group instruction, non-textbook-based instructional materials, or other "current" instructional strategies? Or does the teacher use "traditional" homogeneous instructional strategies that rely primary on text-driven instruction in large groups? Does the teacher explicitly teach problem-solving skills in order to improve student cognitive development?

Factor #5: Are mainstream native-English speakers present in the LEP classroom (e.g., two-way DBE) or not?

As you work with this data collection form, you may add other variables and information that you consider important to collect, to help us define meaningful distinctions in program variations. We encourage your suggestions for improvement of the quality of data collected.

Thomas-Collier Collaborative Research with School Districts: Student Program Participation Data Collection Form

Dear Educator: both Dr. Thomas and Dr. Collier are working with your school district's staff to collect information about the classroom experiences of your students this year. Our purpose is to help your school district to find the most effective instructional practices for your limited English proficient (LEP) or English language learner (ELL) students and your native-English speakers by linking student classroom experiences to long-term student achievement. We appreciate your assistance in gathering data for this research purpose.

School/Class Information

School Year:

_________Name of School: _______________________ School Number
(if used):
___________

School and Class Characteristics

For each statement, please answer
Yes(Y)or No(N)

We have only one type of program for ELLs/LEPs in our school.

 

We have ESL Pullout classes for ELLs/LEPs in our school.

 

We have ESL Content classes for ELLs/LEPs in our school.

 

We have Sheltered ESL classes for ELLs/LEPs in our school.

 

We have ESL Self-contained classes for ELLs/LEPs in our school.

 

We have early-exit Transitional Bilingual classes for ELLs/LEPs in our school.

 

We have late-exit Transitional Bilingual classes for ELLs/LEPs in our school.

 

We have One-way Developmental Bilingual classes for ELLs/LEPs in our school.

 

We have Two-way Developmental Bilingual classes for ELLs/LEPs in our school.

 

All students in our school attend Developmental Bilingual classes.

 

Other important school characteristics? (please list) ______________________________

 

IMPORTANT CONTEXTUAL INFORMATION ABOUT YOUR SCHOOL OR STUDENTS that we should consider in our joint research with your school district:

Teacher Information

Teacher Name: ____________________Teacher Number: (if used) _______________

Teacher/Classroom Characteristics

(circle all responses below that apply)

What grade level(s) are taught by this teacher?

1 2 3 4 5 6 7 8 9 10 11 12

What is the typical class size in your classes?

______ students (please write in class size)

What is the typical number of ELLs/LEPs in your classes?

______ students (please write in number of ELLs/LEPs)

 

Please answer Yes (Y) or No (N) below

Does this teacher team-teach with another teacher?

If yes, name of other teacher? _________________________________________

 

Is there a teacher aide provided in your classes?

 

Other important teacher/classroom characteristic? (please list)________________________________

 
   

Teacher Credentials

Please answer
Yes(Y) or No(N)

Teacher is certified to teach in mainstream classes.

 

Teacher is certified to teach bilingual classes.

 

Teacher is certified for ESL classes.

 

Teacher can teach only in English.

 

Teacher can teach only in another language.
(which language?) ______________________

 

Teacher can teach in English and another language.

 

Teacher is strongly proficient in both English and another language.

 

Teacher is teaching using both languages in instruction this year.

 

ADDITIONAL INFORMATION ABOUT TEACHER CREDENTIALS that should be considered:

Instructional Use of Students' First Language (L1)and English

Please answer
Yes (Y) or No (N)
or Not Applicable(NA)

Teacher teaches ESL as an English class (English as a subject).

 

Teacher teaches ESL through one or two subjects (e.g., math, science).

 

Teacher teaches ESL through all subjects (self contained ESL class).

 

Teacher teaches some subjects through ESL and some through students' L1.

 

In one school week, 100% of instructional time is in English.

 

In one school week, at least 10% of instructional time is in students' L1.

 

In one school week, at least 20% of instructional time is in students' L1.

 

In one school week, at least 30% of instructional time is in students' L1.

 

In one school week, at least 40% of instructional time is in students' L1.

 

In one school week, at least 50% of instructional time is in students' L1.

 

In one school week, at least 60% of instructional time is in students' L1.

 

In one school week, at least 70% of instructional time is in students' L1.

 

In one school week, at least 80% of instructional time is in students' L1.

 

In one school week, at least 90% of instructional time is in students' L1.

 

In a bilingual class, the teacher translates for students.

 

In a bilingual class, the teacher separates the use of the two languages and does not translate.

 

In a bilingual class, the students are allowed to use both languages as needed.

 

Other (please write in and answer Yes or No): ________________________

 
   

Teacher's Commonly-used Instructional Strategies

Please tell us how often each strategy is used by answering:
"Frequently" (F) or
"Sometimes" (S) or
"Not at all" (N)

Teacher uses cooperative learning.

 

Teacher uses microcomputers in instruction.

 

Teacher uses whole language (with phonics included).

 

Teachers uses hands-on instructional materials.

 

Teacher uses discovery learning.

 

Teacher uses critical pedagogy for older students.

 

Teacher uses text-driven instruction.

 

Teacher uses authentic assessment.

 

Teachers teaches learning strategies.

 

Teacher teaches process writing.

 

Teacher uses multicultural literature.

 

Teachers uses pairs and small-group learning.

 

Teacher uses phonics-based basal texts for initial literacy.

 

Teacher uses thematic lessons.

 

Teacher stimulates cognitive development through students' multiple intelligences.

 

Teacher connects curriculum to students' experiences.

 

Teacher uses the community and parents' knowledge regularly as a resource for student learning.

 

Teacher uses visuals, manipulatives, posters, timelines, maps, etc.

 

Teacher integrates art, music, drama into the curriculum.

 

Teacher uses journal writing.

 

Teacher incorporates bicultural knowledge into the curriculum.

 

Teacher mostly lectures (at secondary level).

 

Other: (please write in and answer Yes or No) ________________________

 

Types of Interactions between LEP students and native-English speakers(NESs)

Please answer
Yes (Y) or No (N)
or Not Applicable(NA)

LEPs and NESs interact only at recess and lunch

 

LEPs and NESs interact in mainstream classes taught in English

 

If yes, for how many hours per day? (estimate) ________________

 

LEPs and NESs interact at recess, lunch, and specials (e.g., physical education, art, music, computer lab)

 

If yes, for how many hours per day? (estimate) ________________

 

LEPs and NESs interact all day in all classes in two languages

 

Both we and your school district thank you for your assistance! Please feel free to add any additional comments, questions, or suggestions in the space below. We would especially be interested in any questions (and your answer) that you believe that we should have asked, but didn't.

Back to Main Report