Resources > Case Studies

Arizona Pioneers Statewide Measurement of Students’ Technology Literacy Skills

Results of TechLiteracy Assessment pilot prove valuable and eye-opening.

Breaking new ground, Arizona is the first state in the U.S. to formally measure its students’ proficiencies with technology using TechLiteracy Assessment (TLA) by Learning.com. A spring 2006 pilot program administered Learning.com’s online TLA instrument to more than 24,000 fifth and eighth graders statewide. And from the results, similar numbers of Arizona grade 5 and 8 students will use TechLiteracy Assessment in the 2006-07 school year.

”The data that came from our TLA pilot has just been phenomenal,” says Cathy Poplin, Educational Technology Director at the Arizona Department of Education (ADE), who theorizes that students’ technology skills correlate to overall academic achievement. ”It was fairly easy for us to make the decision to continue it.”

Data to Support Funding Effectiveness
Rather than simply react, Poplin and the department were anticipating what type of data the federal Department of Education (DoE) would next want collected for No Child Left Behind purposes. They were also aware of a clause (Title IID) mandating that all students be technology literate by grade eight by 2006.

Moreover, the state “really had nothing in place that systematically measured anything that we were doing with students using our federal Title IID funds,” says Poplin, referring to Enhancing Education Through Technology (EETT) grants. Thus, to receive their 2005-06 competitive EETT grant funds, Poplin required that districts administer an assessment of technology skills to both students and teachers.

Poplin saw the TechLiteracy Assessment pilot as a “golden opportunity” to begin to proactively collect data. “With the TLA data, we’d be able to show the federal DoE—very concretely— how Arizona was making use of federal Title IID funds to improve student achievement.” At the same time, the same data would be helping to inform districts and the state on how best to deploy funds and resources for technology literacy.

Poplin’s pragmatism—plus “a lot of planning and plain talking”— convinced 50 out of 52 eligible districts in the state to withhold EETT funds in order to participate in the spring pilot. Clearly, the benefits proved compelling. TechLiteracy Assessment “gives us a baseline to work with in terms of gauging our students’ technology literacy,” explains Poplin, “to know if our students are even getting close to proficiency.” From that, adjustments to instruction or professional development or other areas can be made.

In this age of K-12 accountability, data packs real clout. “We felt that TechLiteracy Assessment’s data would give us a lot of leverage—with our state legislature, with our districts and the communities, and within our [state] department [of education] itself,” says Poplin. And, she adds, “it has done all of that for us so far.”

Authentic Assessment Engages Students to Point of Quiet
Arizona schools administered TechLiteracy Assessment over a span of about seven weeks. Students used their school’s computer labs to take the online test during a single class period. By all accounts, it was an easy process for all.

Teachers who proctored the assessment noted that students seemed “very engaged” by the online TLA test. One responded that she would’ve loved to have had a video camera with her to document the phenomenon. Another mentioned how amazingly quiet the computer lab was during the assessment.

When taking the online test, students interact with assessment content in ways that allow them to demonstrate their proficiencies. Often, they must perform actions via simulations, rather than pick answers from among multiple choices. Thus, students must be able to format a paragraph, apply a spreadsheet formula, or conduct a database search. And they must demonstrate durable skills via generic menus and commands, not through brand-specific memorized shortcuts.

Students’ internal motivations must be factored in to their engagement as well. Students place great “personal value” in technology, observes Mary Knight, who proctored TLA in her district, Flagstaff Unified. “Technology is such a big part of their everyday lives and world,” she explains, that “our students were excited at the opportunity to be assessed in this area.” In other words, students wanted to be tested on their proficiencies with technology applications. They also wanted to know their scores immediately. Even though it was a pilot, Learning.com provided the individual student reports for teachers to review with students before the end of the school year. All 50 districts that participated also received aggregated reports to share with administrators, teachers, technology coordinators, and the community.

Results of TechLiteracy Assessment Spark Discussion, Lead to Changes
Results from the TLA pilot were eye opening. They certainly have sparked discussions. Some districts that expected their students to perform well, for example, were surprised to learn that they did not score all that well. Other districts, such as one that draws a majority of its students from Navajo reservations in which there isn’t even electricity going to most homes, scored higher than the district expected.

Statewide, 63 percent of the nearly 12,000 Arizona eighth graders tested by TLA did not meet proficiency in technology skills. Of the fifth graders tested, 73 percent were not proficient. “It told us we have work to do, certainly” says Poplin. “More importantly, it showed us some weak spots, which will help us better target our efforts. That’s the key.”

And districts are doing just that. Washington Elementary School District will be adjusting its junior high curriculum based on the TLA pilot’s data. Arizona’s largest pure K-8 district, it serves 25,000 students from Phoenix and Glendale in 32 schools. Mike Cannon, technology training coordinator for the district, explains that due to really tight schedules, a lot of technology instruction occurs in grade 7. After that, focus tends to fade. This pacing may be problematic, according to TLA results from the district’s 450 tested students. “We learned that our kids aren’t retaining as much of their technology skills as we would like,” says Cannon. Thus, he says, discussions are now underway with the junior high schools’ computer teachers on how to reconfigure the technology curriculum in grade 8 to better maintain mastery levels.

Direct Instruction May Matter
The small Wilson School District #7, with just two schools, also participated in the TechLiteracy Assessment pilot. It tested 300 of its 1,500 preK-8 students. But Wilson administrators were not pleased with the results. “It was a surprise to us,” admits curriculum director Debra Turl, “that our kids didn’t do very well on the TLA test.” After all, Wilson has offered a 1:1 ratio of computers to students for years.

Located near Phoenix’s Van Buren airport, weekly hotels and two shelters are among those that send students to Wilson’s Primary (K-3) or Elementary (grades 4-8) schools. It’s one of the poorest districts in Arizona, but its classrooms are high-tech and have been since 1993. Currently, every student’s desk houses a computer and monitor under glass. The 1:1 ratio “lets us meet the needs of every student at an individual level,” explains Turl. “Everybody gets their curriculum through the computers, but they also get remediation or enrichment or whatever they need.”

However, using computers to receive and study curricula does not automatically translate to technology literacy, the TLA pilot clearly indicates. “We’ve learned that, even with a 1:1 computer to student ratio, you can’t just assume that students are going to learn technology skills,” says Turl.

The district is already thinking about how to use the TLA data to change instruction and improve proficiencies. Turl explains that one takeaway from the pilot was that their students need to use technology applications more frequently. Moreover, teachers had long been saying that students needed more keyboarding practice. Now, backed by the TLA pilot, it’s obvious that time must be found somehow and set aside to teach and apply these skills.

“We believe in direct instruction in reading and math,” observes Turl. “So now it seems we need to have it—at some level, at least—for technology skills as well.”

Teacher Training Does Matter
Holbrook Unified School District #3, on the other hand, performed better than expected in the TechLiteracy Assessment pilot. In testing about 240 of its fifth and eighth graders, its junior high school received one of the highest scores in the state.

Tucked in northeast Arizona, many of the nearly 2,100 students attending its five K-12 schools come from Navajo reservations. While remote and rural, the district is also “progressive and proactive” with technology, says Ann Gardner, its technology coordinator. All Holbrook schools have computer labs, most classrooms have extra computers, and digital whiteboards are prevalent too. Additionally, Holbrook has a technology curriculum in place.

But the pivotal clue to its students’ relatively high scores on TLA, the ADE’s Poplin feels, lies in the heavy emphasis Holbrook Unified places on professional development for its teachers. “These scores are testimonials for a district in which a whole lot of things are being done right,” says Poplin. “All of our professional development for teachers focuses on technology integration,” agrees Gardner. “Actually, it focuses on good lesson design because technology is nothing without effective practices to support technology as a tool.”

As the TLA pilot scores reveal, the district’s high priority on training teachers in technology integration is paying off. “Technology is becoming embedded into teachers’ instruction, their thinking, and their planning,” Gardner says of the teachers.

TLA to be continued for 2006-07
With the data from the TechLiteracy Assessment pilot in hand, Arizona is poised to improve technology instruction and meet national and state mandates. So it’s no wonder that everyone involved found the TLA pilot to be well worth it and most districts are participating again in 2006-07, during which both a pre-test and post-test TLA will be administered to fifth and eighth grade students statewide. In fact, Arizona is continuing with TechLiteracy Assessment despite having their federal EETT Title IID funds cut by 50 percent or more this year.

If the whispers prove true, and technology skills are the next curricular area to be formally tracked by federal accountability measures, Arizona will be more ready than most to meet the new challenge.

“The data has just been invaluable,” says Poplin. “Upper management here at the Department of Education all agree on that.” So, too, do the individual districts that participated in the pilot, where teachers, students, and administrators are all eager to continue curriculum enhancements and professional development efforts to improve students’ technology learning.

Share: Reddit | Digg | del.icio.us | Google | Yahoo | What is this?