Outcome Tracking: Practical Tips

In this resource brief, community colleges in Civic Ventures’ Encore College Initiative share tips for tracking student outcomes in programs for people 50 and older seeking encore careers in education, healthcare, social services and the emerging green economy.

Outcome tracking is a hot topic across higher education today. Federal, state and local government, accrediting agencies and private foundation funders are asking for evidence that college education and training is not only accessible, but effectively preparing students with marketable skills employers need. Increasingly, stakeholders expect to know if degree and certificate holders are successfully finding employment in their chosen fields.

The most important stakeholder in the outcome tracking arena is the community college itself. Community colleges can test how new programs are working and how they can be improved. Outcome data also provides evidence that helps attract new students and new sources of support for growth.

The tips below for community college administrators provide guidance for developing an effective approach to measuring the success of encore college programs.

  1. Embrace outcome tracking for your own reasons.

  2. It’s best to develop an outcome tracking system for your encore college program because you understand its ultimate value to your program and your students’ goals. It’s true, your data collection also needs to show progress toward your original program goals submitted to funders, but in the ideal situation, these goals are already coordinated with one another.

    The overall design of your outcome tracking system should collect information in a feedback loop process that allows you to tweak existing programs or add new ones. A new report from the Bridgespan Group, Measurement as Learning, discusses how to use data to adapt programs quickly and respond effectively to current circumstances. Measurement also helps with resource allocation – surveys of students and graduates show where additional money is needed (such as training in job search skills) and where it is not.

  3. Develop a systemic approach.

  4. College administrators involved with tracking encore students have reported a variety of issues to consider as you develop your organization’s strategy. The most common refrain is “It’s difficult to stay in contact with former students.” Colleges report mixed success, finding pockets of learners who are easy to track but estimating great difficulty reaching about one third of the students who have graduated. A more candid response from administrators is frequently, “In general, we don’t have a system.”

    Elizabeth Johnston O’Connor, Director of Institutional Research and Effectiveness at Cape Cod Community College, advises taking a systemic approach from the very beginning. “Start by using the college’s existing system for identifying and collecting basic student information,” she suggests. To track plus-50 students, Cape Cod Community College’s program staff uses an electronic reporting program to track each student’s contact with the college. If a student seeks out support services, the staff can access the students’ information and see what courses they are taking and where they need help. The information is then incorporated into the advising process. When they exit the college, tracking becomes the responsibility of O’Connor.

    Be thoughtful about what you want to learn from your data collection and design your instrument accordingly. Build all requirements into your tracking system from the beginning. For funders, be sure you can track the outcome measures you specified in your proposal. (It’s much easier to build in tracking for your promised outcome measures from the beginning than to spend additional time on the back-end searching for that information.)

    O’Connor cautions, “Ask as few questions and ask them as directly as possible.” You’re not finished designing a systemic approach until you identify effective ways to contact past students, find a user-friendly way to collect the data, house student information in a secure environment, and make and implement a plan to analyze data and report outcomes.

    When do you send out a survey to former students? Some hoping to report results promptly to a funder found that after one month, most encore students said they were “still looking for a job.” O’Connor says that the industry standard for follow-up surveys is six months. Grantees should communicate these expectations with funders to decide the most appropriate timing for follow-up.

  5. Be sure to collect information specific to encore careers.

  6. For encore college programs, these are the key outcome questions:

    • Is the student employed in an encore field? In what role and in what organization?
    • How satisfied is the employer with the encore employee’s education and skills?
    • What kind of social impact has the encore employee made (or is expecting to make)? From the employer’s perspective? From the employee’s perspective?
  7. Use electronic media tools.

  8. Don’t be afraid to use electronic media with the plus-50 cohort. Colleges that have used email to contact graduates have had promising successes. Others have gone beyond email. Jan Davie, Director of the Center for Workforce Transition at Gateway Community College in Phoenix, Ariz., has recently begun using the online survey tool Survey Monkey, which has had the highest response rate to date compared to other surveys used in the past. But Davie adds one caveat that is emphasized in the next tip.

  9. Stress the importance of follow-up from the beginning – with students and faculty – and let them know that program quality and continued funding may rest on their participation.

  10. According to Rosemary Dillon, recently retired Interim Dean of Academic and Student Affairs in the Division of Health Sciences, Social and Behavioral Sciences and Human Services at Cape Cod Community College, “It’s very important that students hear when they enter the college the reasons why we need information.” Once students realize that their feedback is integral to the enhancement of programs and in reporting to funders, they are more likely to respond to follow-up inquiries.

    Make sure students are prepped so they know that tracking results will be part of their experience while enrolled and after leaving the college. Encore programs that set and stress this expectation to students have had greater success with response rates. O’Connor cautions that if you don’t set expectations, your response rate is likely to be in the single digits, which won’t be reliable and cannot be used.

    “At Gateway we built our request on boomers’ desire to have meaning and leave a legacy – so we position our surveys to tap into that need,” Davie said. “We say that they can help create a better program.”

    Dillon adds, “Actually, it’s especially important to get the faculty on board, too, because we often need to use instructional time to have students complete surveys.” And informed faculty can further stress the importance of responding to college inquiries.

  11. Learn from and use your college’s data collection efforts on student outcomes, particularly when they are required by outside organizations.

  12. Some encore college programs partner with local SCSEP (Senior Community Service Employment Program) agencies. SCSEP is a federal program authorized by the Older Americans Act that trains low-income adults 55 and older to develop new job skills with the ultimate goal of moving 30 percent of trainees into unsubsidized employment. The local agency works with the college to identify prospective and eligible students to receive the subsidized training and field placements.

    Davie, who has four years of experience tracking encore graduates of the Caregiving and Human Services Certificate Programs at Gateway Community College, has found that higher skilled SCSEP participants do get fulltime employment faster than lower skilled individuals who tend to remain in the program longer for more training and field experiences.

    Because legislators and other funding agencies often require colleges to track students, your college may already have in place some systems and practices that you can use, piggy-back on or adapt for your encore programs. For instance, those overseeing a college’s Federal Perkins Loan Program are expected to track grant recipients one year after leaving college.

    At Cape Cod Community College, the Perkins survey request that goes out a year after graduation is sent from the president of the college, which is noticed by recipients. The program completes two mailings – a first mailing, then a second if no response, and if those fail to produce a response, an email blast. The most effective technique to improve completion rates has been phone calls to those who haven’t responded, Davie reports. Once on the phone, it’s even possible for the student to complete the survey verbally. Unfortunately, phone calls can be very labor intensive and expensive. Even so, one Cape Cod staff member commented, “Some things work, and some things don’t. We try everything.”

  13. Involve encore employers in your data collection and feedback system.

  14. Dillon notes that for some Cape Cod Community College health care programs to remain accredited, the college must conduct surveys of local employers. According to Dillon, the college gets a good response from employers because it has developed quality working relationships with them. Says Dillon, “We know where they live!”

    Employers are an excellent source of information for other encore programs, too. Gateway Community College relies on employers through the advisory boards they use to keep the program “in the know.” Davie says the college also gets input from Gateway’s health care program, the Department of Aging, and a random sample of local employers. She emphasizes the importance of having employers on board and helping with student tracking and placements. Other encore program directors have had similar partnership agreements where employers gave regular feedback on the number of placements and the quality of job readiness.

To see more resources for encore college planners, click here.

To learn more about the Encore College Initiative, click here.