Summary
As a federally funded AmeriCorps program with more than 270 tutors across the state (350 for the 2007-08 program year) and just 16 staff members available to manage the program, America Reads – Mississippi (ARM) is always seeking improved and more efficient ways to: · Get tutors the ongoing training and guidance they need to address their individual goals and challenges with each student. · Facilitate the development and ongoing growth of a learning community among current and past tutors in the program, regardless of their existing relationships with one another. · Build and access institutional memory across program years of which tutoring techniques and strategies work at each of its 59 tutoring sites (80 sites for Program Year 2007-08). · Know whether its program is on target with tutor training and programmatic goals at any moment so that if it is not, regional and central office staff can make immediate changes to get back on target before it's too late. · Decrease the time and money staff spends on tutor monitoring, data collection and reporting, and use that freed up time and money to directly enhance tutor competencies and other aspects of the program. · Create well written reports that carry significant weight with reliable and valid data for funding sustainability. · Spend only a minimal amount of time training tutors and staff members to use any adopted process. This case study documents the reasons ARM decided to begin using the America Learns National Performance Measurement & Support Network in August 2004 to accomplish the above goals and to ultimately provide more valuable services to Mississippi children. The document then tracks the returns of ARM’s investment in the Network over the past three years. The study is organized into the following sections: - ARM Program Overview & Goals Leading to Network Adoption
- Strategy, Adoption & Implementation
- The Research Driving the Network
- Managing and Measuring the Network
- Lessons Learned
- - - - - - - - - - - - - - - - - - - - - - - - - - - - Section 1: ARM Program Overview & Goals Leading to Network Adoption About America Reads - Mississippi
Since 1998, ARM has been working to simultaneously address the massive inequities in Mississippi public schools, the major teacher shortage in its state, and the enormous group of Mississippi adults seeking gainful employment[i]. By hiring and training low income (or zero income) individuals to teach the state’s most struggling students to read, ARM not only gives Mississippi children an improved chance at academic and life success, but also reduces the demands on public welfare assistance by employing adults for up to two years and motivating the best performers to further their own education and become the state’s next cohort of top-notch teachers. Since the program’s inception, 75% percent of ARM’s 1,910 AmeriCorps members (its tutors) have reported that they intend to become certified teachers. AmeriCorps, the federal program funding ARM, provides tutors with $4,725 for each year of service that can be used to fund higher education and pay back student loans. AmeriCorps also awards members a small monthly stipend that they use to support themselves during their term of service.
Mississippi’s children not only benefit from ARM’s work in the short term by receiving personalized tutoring and mentoring services to help them hurdle the historical, cultural, and social challenges they face, but also gain by having better trained teachers in the classroom and communities with more consistently employed adults (leading to a number of positive outcomes, including more stable households and neighborhoods).
Programmatic Challenges
A number of challenges work against ARM’s efforts to ensure that its tutors have the competencies to deliver high quality services and that its managers can track, evaluate and respond to tutors and students' needs over the short and long term. A number of factors further complicate these challenges: · Tutors’ education and professional backgrounds: Most ARM tutors are not child development experts and have not had professional experience as reading tutors. Thus, the program must provide a significant amount of initial and ongoing training to its tutors throughout the year that promotes a steep (almost instantaneous) learning curve. Given everything the tutors must learn – content and pedagogical knowledge – providing a cornucopia of general trainings that aren’t aligned with any one individual tutor or student’s needs is insufficient. While ARM’s regular, in-person monthly trainings address issues that all tutors benefit from greatly, finding a way to provide individualized training and support between those trainings is just as essential. · Geography: ARM tutors are spread across the state. Many of them live and work 50 miles or more from the closest full-time staff person. ARM administrators believed that in order to make the impact the organization was aiming for, the program needed to find a way to overcome this barrier and to begin tracking and quickly responding to each individual tutor's (and therefore, each student’s) goals and challenges with rapid, targeted support. The program also wanted a way to facilitate ongoing sharing and learning among the tutors despite the distances between many of them. · Staff workloads and stress: ARM’s 16 staff members have huge workloads consisting of human resources administration (including payroll), accounting, tutor training, data collection, reporting, newsletter writing, other public relations work, and more. Any new process or solution the program implements must never tax its team even further. · Technology access and familiarity: Many ARM tutors do not have much experience with computers or the Internet. If the program decided to pursue any solution that relied on Internet use, ARM needed to ensure that its tutors could learn to use it with minimal training and almost no support from overburdened staff members. ARM’s Business Process Goals
ARM’s central office administrators continually look to bring efficiencies to the program. "The Network is giving our AmeriCorps members the immediate, personalized technical assistance they need while providing regional and central office staff with instant feedback on how well the program is operating at each region, information about how members are meeting program goals, and what staff need to do to improve our in-person training efforts." Randee Williams
State Director |
Specifically, the program always looks for ways to: · Get tutors the ongoing training and guidance they need to address their individual goals and challenges with each student. · Facilitate the development and ongoing growth of a learning community among current and past tutors in the program, regardless of their existing relationships with one another. · Build and access institutional memory across program years of which tutoring techniques and strategies work at each of its 59 tutoring sites (80 sites for Program Year 2007-08). · Know whether its program is on target with tutor training and programmatic goals at any moment so that if it is not, regional and central office staff can make immediate changes to get back on target before it's too late. · Decrease the time and money staff spends on tutor monitoring, data collection and reporting, and use that freed up time and money to directly enhance tutor competencies and other aspects of the program. · Create well written reports that carry significant weight with reliable and valid data for funding sustainability. · Spend only a minimal amount of time training tutors and staff members to use any adopted process. While ARM adopted a number of effective policies and procedures over the years to help address these goals, the challenges detailed above still led to undesirable amounts of time and money being spent to realize the program’s mission.- - - - - - - - - - - - - - - - - - - - - - - - - - - - Section 2: Strategy, Adoption & Implementation Strategy for Meeting its Goals
While ARM had the goals listed above for some time, the program did not believe it was possible to address all of them in a simple way until a colleague passed along an e-mail she received in July 2004 about a new service called the America Learns National Performance Measurement & Support Network ("the Network"). ARM decided to review the Network two times via a Web-based tour led by an America Learns team member.
Examining the Network
As ARM’s central office administrators examined the Network, they made sure it would be cost effective, could be customized to fit with the many facets of its program[ii], would work quickly over old computers and dial-up modems, would not disrupt tutors and staff members’ personal and professional daily routines, would be easy to use by people unfamiliar with computers, and would be accurate in getting tutors and managers the training, guidance, support and data they needed right away.
ARM also confirmed that its staff would have quick access to technical assistance when needed – both support built into the application and via phone and e-mail.
Administrators further ensured that its staff would not spend a lot of time generating and uploading new tutor training content to the Network. Because the America Learns Network came with high quality content designed just for reading tutors, and because America Learns team members regularly analyze the challenges faced by its clients’ tutors and then create new strategies to overcome those challenges, ARM staff did not have to spend time populating a database of training content from scratch. When ARM’s central office staff saw the Network’s potential to meet its goals and had colleagues review it, ARM’s strategy quickly became one of amending its budget to find funding to pilot it. ARM made this happen in about three weeks[iii]. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Section 3: The Research Driving the Network
The R&D Process
The America Learns Network was built upon hundreds of interviews with tutoring and mentoring program staff members and their tutors and mentors. Much of that research involved asking these individuals whether they even wanted to receive ongoing training and support, and if so, what types of training and support they would make time to take advantage of.
Tutors and mentors' responses focused more around content delivery systems rather than content itself, and most of the responses were not "I want to use X" expressions such as, “I want to use a social networking and bulletin board website.” Most responses were of the type, “I’ll use anything except X.” People ruled out generic resource websites (imagine something called “tutorsupport.org” that contains a tutoring strategies database), e-mail listservs, discussion boards, guidebooks, online videos, videotapes, and additional in-person workshops. The main reasons for those media not being attractive was that they are simply too time consuming to use, and that when tutors and mentors do use them, they usually do not find the information they need.
Interviews with program staff members nationwide revealed that they not only wanted to provide support to their tutors and mentors, but also wanted a way to get a better picture of what exactly their tutors and mentors were doing with students on a day-to-day or week-to-week basis so that they could design training opportunities that were aligned with those actual needs. When staff provided that support, they didn’t want to deliver it weeks or months after their tutors and mentors needed it.
Staff members also cried out for ways to reduce the amount of time spent on data collection, reporting and evaluation. In recent years, grant makers have significantly increased the amount and types of performance measurement and evaluation data that grantees must report to retain and renew funding. This culture shift has led people hired to manage programs to become “accidental researchers.” According to Randee Williams, ARM’s State Director and long-time resident of the AmeriCorps community, “These individuals’ limited knowledge, skills, and experience in this area jeopardize future eligibility for grant awards.” In short, through the process of collecting the data ARM needs, ARM is also able to provide individualized support to its tutors. |
The Basics of How it Works
The Web-based, patent pending Network allows ARM to fulfill its needs of collecting important data and best tutoring strategies coming out of its program, sharing those practices when tutors need them, and measuring tutor impact in a single step – the completion of a survey form, journal or reflection log. Unique aspects of the Network also allow ARM tutors to share their strategies with and access the best strategies of other educators using the Network worldwide. ARM is able to accomplish these goals without using media its tutors usually lack time to use on a regular basis – e-mail lists, web-based forums, bulletin boards, formal courseware, blogs, wikis or guidebooks.
In short, through the process of collecting the reporting and measurement data ARM needs, ARM is also able to provide individualized support to its tutors. The Network’s impact on strengthening and increasing tutor competencies extends into the physical world as well, as ARM uses the data it collects to design in-person training workshops, one-on-one and small group check-ins, and paper-based newsletters to make those forms of training and support more relevant to tutors’ daily needs. As discussed more Since the tutors are able to use the Network to give its staff the tracking, monitoring and evaluation data ARM needs, staff members are able to provide the tutors with the timely training, guidance and support they need.
Launching the Network
Year 1 Training (Pilot Year)
During its first year using the Network, ARM conducted live demonstrations of it for all of its tutors with an emphasis on the reasons ARM invested in the Network for them. The main benefits ARM discussed included: 1. Providing tutors with timely, specific, step-by-step strategies to use that are related to their own goals and challenges; 2. Giving tutors an opportunity to share their own successes with others in their program and with other educators using the Network across the nation; 3. Helping tutors to stay focused and being more accountable to themselves and to their students; and 4. Allowing their program managers to be more responsive to them. ARM required its Team Leaders to participate in the pilot and also recruited additional volunteer participants (for a total of 100 users)[iv]. Because the Network is so easy to use and because all of its tutors are familiar with the process of completing surveys, ARM found that its training needed to last one hour. America Learns reduced ARM’s training preparation workloads by providing materials to use and hand out during the trainings.
Years 2 and 3 Training
ARM provided similar training and support during its second and third years using the Network when it increased the number of tutors using it to 150.
Year 1 Ongoing Culture Integration
ARM called the tutors piloting the America Learns Network "America Learns-ARM Members" and gave them special "happies" (paper weights in the shape of apples, memo note holders) throughout the year to thank them for their participation. (ARM also provided awards to these members during its end-of-year celebration.) Further, ARM regularly shared the data generated by the Network with its tutors to demonstrate how ARM was using it to improve programming and to report to the Corporation for National Community Service (the federal agency that funds the program). Given the numerous demands and limited financial awards its tutors receive, ARM administrators believe that their tutors must know that their time and effort is valuable and is benefiting not only themselves but the program as a whole.
Years 2 and 3 Culture Integration
In addition to the activities above, ARM continued to market the Network internally during its second and third years by celebrating an author of each region's "tutoring strategy of the month" (the best strategy submitted by a tutor via the America Learns Network), giving awards to those tutors at regional meetings. (Each ARM region selected a winner.) From the regional winners, the central office administrators selected the winning state tutoring strategy of the month and highlighted this person in its monthly newsletter and gave the person an engraved plaque. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Section 4: Managing & Measuring the Network How ARM Manages the Network
Weekly, Personalized Survey Completion Reminders
The Network sends reminder e-mails to all of its tutors each week, asking those who have not yet completed a survey to do so. The e-mails act as "automated managers" for ARM.
Since implementing its plan, ARM's weekly survey completion rate average has skyrocketed from 64% to over 90%, a completion rate rarely achieved in the social services sector. |
Monitoring Survey Completion Rates & Following Up
ARM regional staff members monitor survey completion rates within their regions and central office administrators monitor the state as a whole. Staff take advantage of the Network's easy to use features designed explicitly for following up with those tutors who do not complete surveys to learn the reasons they did not do so, and to remind tutors that completing the surveys is not only a privilege, but a requirement for all America Learns-ARM Members. ARM worked closely with America Learns staff to develop an internal Network management plan so that ARM could build the process of staying on top of this information into its culture. Since implementing its plan, ARM’s weekly survey completion rate average skyrocketed from 64% (2004-05) to 77% (2005-06) to over 90 percent (2006-07), a completion rate rarely achieved in the social services sector.
How ARM Measures the Impact the Network is Making
Use of Recommended Strategies by Tutors
It's essential that tutors learn from the Network and apply that learning in their tutoring situations. Since ARM doesn’t have the manpower to observe the tutors in action more than a few days a year, ARM relies upon tutors’ reports via the Network to learn whether tutors are using the strategies. During the 2005-06 survey, out of 337 surveys completed over a number of periods, one hundred percent of members who reported that the Network had ever recommended a strategy to them following the completion of the survey also reported that they had found at least one recommendation to be helpful. When asked to explain the reasons they use the strategies, tutors wrote responses such as, "They were very simple to follow and easy to understand", "Because I ran out of ideas and needed more new ideas," "The strategies help in every possible way with the students," and "The strategies helped because [they] gave me a new way to teach something, therefore making it more interesting to me and to the students." "The strategies helped because [they] gave me a new way to teach something, therefore making it more interesting to me and to the students."
ARM AmeriCorps member |
Based on this and other data, ARM has learned that it needs to add a number of resources around non-tutoring issues that its tutors struggle with as AmeriCorps members. All AmeriCorps members, for example, must recruit volunteers in their community, and many must participate in homeland security and emergency preparedness activities. (For program year 2007-08, selected members will also perform hurricane recovery service and others will mentor children of incarcerated parents.) Because America Learns' current content expertise does not cover these issues, ARM will need to invest time to create its own content and make specific requests of its tutors to submit their successes in these areas.
ARM also surveys tutors around how supported they feel and what else they need from staff to accomplish their mission.
Use of Data by Regional and Central Office Staff, and Resulting Financial Returns
Aside from using survey data to improve training and overall programming, ARM also uses it for reports and evaluation. The Network has reduced the amount of time ARM administrators and staff members spend on annual reporting by more than 660 hours (four weeks’ worth of time) – translating into an annual dollar savings of more than $8,000 once its financial investment in the Network is taken into consideration. (Note that this figure only includes savings the Network brings to ARM around data collection, performance measurement, reporting and evaluation, not tutor training and support.) ARM has been able to reallocate the inordinate amount of time its staff members used to spend collecting and reporting data to using data to provide enhanced in-person training and support to its tutors. While the accompanying staff stress level reduction is tough to measure, administrators report that “It's priceless.”
ARM also uses the data to shape significant portions of its in-person training workshops, making those trainings more meaningful to its tutors. The program sometimes uses strategies from America Learns to shape its workshops. As an example, following is a portion of an e-mail sent to America Learns from an ARM administrator in response to ARM staff finding that its tutors were running into a number of classroom management challenges: “…we are planning to include classroom management to our training curriculum and offer alternatives/strategies that our members can use in certain situations. We will definitely take advantage of the America Learns strategies.” On a four point scale (Poor, Fair, Good, Excellent), more than 90% of ARM tutors using the America Learns Network rate in-person trainings as “Good” or “Excellent”.
ARM has been able to reallocate the inordinate amount of time its staff members used to spend collecting and reporting data to using data to provide enhanced in-person training and support to its tutors. While the accompanying staff stress level reduction is tough to measure, administrators report that “It's priceless.” |
Randee Williams notes, “The Network is giving our AmeriCorps members the immediate, personalized technical assistance they need while providing regional and central office staff with instant feedback on how well the program is operating at each region, information about how members are meeting program goals, and what staff need to do to improve our in-person training efforts. I can pull up this information from any Internet connection 24/7.”
Further, ARM’s use of the Network is making its quarterly reporting to government funders a breeze. While regional staff used to have to compile data and send it separately to the central office, central office administrators now pull up and copy the data from the Network into their reports and into WBRS, the web-based reporting system managed by the Corporation for National & Community Service. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Section 5: Lessons Learned Following is a list of some of the most important lessons ARM has learned since beginning to use the Network: "The worldwide collaboration the America Learns Network facilitates is what the national service and education worlds have been waiting for. The fact that we can easily couple these benefits with reporting, performance measurement and evaluation makes the Network a must have for us." Randee Williams
State Director |
1. Technology is available to help ARM accomplish its goals listed above. The Network is allowing ARM to improve the delivery of services to Mississippi children. 2. Develop and implement a formal management plan to implement technology such as the Network. During its first year using the Network, ARM did not make enough time to really build it into its culture. Since then, ARM has worked with America Learns to think through each action that needs to happen to effectively capitalize on the Network's intended and unintended benefits. (America Learns now uses an adaptation of the management plan ARM developed with all of the other organizations it serves to ensure that those organizations effectively build the Network deeply into their cultures. 3. It’s important that the tutors feel that their opinions matter and that they are an integral part of the program’s evaluation process and improvement plans. The tutors are “in the trenches,” and ARM believes that that reality must be acknowledged and respected. The Network offers the necessary vehicle for ARM’s program staff to constantly learn about and then respond quickly to tutors’ experiences and opinions. 4. When ARM provides its tutors with ongoing training resources that respect tutors’ time constraints, tutors are far more likely to use those resources. By incorporating reporting, evaluation, strategy sharing and ongoing tutor support into one step – the completion of a survey – ARM has found an answer to getting much of what it needs from tutors and providing what it needs to provide to them throughout the year. 5. It’s essential to train tutors around what a well-written strategy looks like. As mentioned earlier, ARM uses the Network to constantly collect and make use of the best strategies submitted by its tutors. ARM found that it has to set clear expectations around what these strategies look like so that when tutors submit them, they’re sufficiently detailed for others. While ARM still has a ways to go in this area (much of its success hinders on its tutors’ writing abilities), ARM has made substantial progress in getting its tutors to submit higher quality strategies. As a result, America Learns highlighted two of its members’ strategies as National Strategies of the Month during the 2006-07 school year, an honor bestowed to only twelve tutors each year. 6. Capture and use the amazing strategies and activities its tutors are creating and sharing via the Network. Until ARM began using the Network, its tutors' original, innovative strategies were lost. ARM collected some success stories on paper, but the program did not have the ability or capacity to organize and then distribute those resources to tutors when they needed them. Now, when tutors complete their weekly surveys, they instantly receive tips and resources from their colleagues statewide -- even from colleagues who are no longer with the program. And since the program is connected via the Network to other programs America Learns serves nationwide, the strategies its tutors are creating are helping other students across the country in programs ARM does not formally interact with. Randee Williams notes, “With the Network, it’s now possible to create a culture that values and facilitates ongoing sharing and learning among tutors, regardless of their physical location and relationship with other tutors in a community – in this case, the ARM and America Learns Network communities. The whole notion of ‘Think globally, act locally,’ has been rearranged through this process. ARM and all other organizations using the America Learns Network are thinking locally, acting locally, and making a global impact simultaneously. The worldwide collaboration the America Learns Network facilitates is what the national service and education worlds have been waiting for. The fact that we can easily couple these benefits with reporting, performance measurement and evaluation makes the Network a must have for us.”
Endnotes
[i (back)] Mississippi’s unemployment rate is the highest in the nation at 6.9% (SOURCE: bls.gov; April 20, 2007) [iii (back)] Rather than budgeting the Network as a separate line item, ARM found it easier to justify its use of the Network to federal grant officers when the program budgeted it under line items dealing with AmeriCorps member development and program evaluation.
[iv (back)] Team Leaders are AmeriCorps members who receive specialized leadership training so that they can effectively assume additional responsibilities that help to successfully implement the ARM’s programming at each school. These members do not receive additional financial compensation. |
