Data-driven solutions
for mission-driven organizations

Process & Outcome Evaluations of a Multifaceted Child Care Improvement Effort

Child Care Solutions is a not-for-profit organization founded in 1975 to provide information and resources to parents, providers, employers, and community leaders in Onondaga and Cayuga Counties (NY) who seek solutions to their child care needs. In partnership with the Early Childhood Alliance, Child Care Solutions has been working tirelessly to assure child care options are not only available but of high quality.

Beginning in 2021, The Child Care Quality Improvement Pilot (CCQI) funded by Onondaga County, provided extra resources to this effort. This year the pilot’s focus is on family child care. Key components include training family care providers to implement an evidenced-based curriculum (The Creative Curriculum), one-on-one coaching in using assessments (a curriculum fidelity tool, the Family Child Care Environment Rating Scale [FCCRS], and the Ages and Stages questionnaire) to inform lesson planning, and communication with parents. There are also stipends for the providers.

As the independent evaluator of the pilot, A&O’s work will have two key components. First, a process evaluation. This is designed to assess CCQI’s progress implementing the project components they’ve planned.

Second, an outcome evaluation will measure progress toward the intended outcomes and describe the activities’ early impacts. Stakeholder feedback and assessment data will be important to that evaluation, and A&O will synthesize data from a variety of sources to formulate suggestions for improvement.

Child Care Solutions of Onondaga and Cayuga Counties (CCS): A Child Care Needs Assessment

In the past, Apter & O’Connor has worked with Child Care Solutions several times to support their needs assessment related to the current state of child care. A&O designed specific inquiries to examine child care supply and demand. The final product illustrated the child care needs unique to these geographic areas and the related work-life issues for both child care providers and the families they serve. The assessments included: a detailed review of national, state and local data; focus groups with child care providers, local government officials, funders and human service experts; surveys for parents and providers; and interviews with human resource professionals.  The rigorous methodology employed for each study provided child care advocates with a powerful tool to design new interventions and educate state and local legislators and funders.

The Family Reading Partnership is a coalition in Ithaca, NY, comprised of individuals, businesses, schools, libraries, and other organizations acting to “create a culture of literacy” to assure all children have early, frequent, and pleasurable experiences with books, as a special part of everyday family life, right from the start!  A&O worked with the Partners to build evaluation capacity, guide a literature review, and develop programmatic logic models. A&O conducted external evaluations of three programs and with a participatory1 approach built the Partnership’s capacity to continue their evaluation work internally.

 [1] Whitmore. E (Ed) (1998).  Special Issue:  Understanding and Practicing Participatory Evaluation.  New Directions in Evaluation.  Vol. 80, pp. 1-99

21st Century Community Learning Centers (21st CCLC) and the Extended Day/School Violence Prevention (ED/SVP) programs

A&O conducts evaluations for several 21st Century Community Learning Centers (21st CCLC) and the Extended Day/School Violence Prevention (ED/SVP) programs. These programs are federally, and state funded to support tutoring, academic enrichment and a broad array of youth development opportunities before and after school to complement the school’s regular academic program. It features partnerships with school districts and community-based organizations to programs target students from high-poverty neighborhoods and attending low-performing schools.

A&O’s evaluation activities include systematic observations at all sites, interviews with staff and administrators, surveys of all stakeholder (students, staff and parents) and extensive analysis of student academic data. A&O has worked with large multi-site school districts as well as smaller community agency sites, including The Syracuse City School District, Watervliet Central School District, The YMCA of Central New York, and the Center for Community Alternatives.

Program Directors consistently note the value of these rigorous evaluations and the ongoing feedback and technical assistance for program improvement that the A&O team provides. These independent evaluations have provided the needed evidence and critical information to convince legislators at all levels of government to continue funding these after school programs.

The Wayne County Task Force for Run Away and Homeless Youth and The Wayne County Housing Consortium

Apter & O’Connor developed and analyzed a survey to 792 eighth and eleventh grade students who attended 6 schools in Wayne County a rural community in Western New York.  The survey gathered information related to the prevalence and issues surrounding homeless and/or run away youth. The survey design probed sensitive information in a manner appropriate to young people of diverse backgrounds.

21st Century Community Learning Centers (21st CCLC) and the Extended Day/School Violence Prevention (ED/SVP) programs

A&O conducts evaluations for several 21st Century Community Learning Centers (21st CCLC) and the Extended Day/School Violence Prevention (ED/SVP) programs. These programs are federally, joand state funded to support tutoring, academic enrichment and a broad array of youth development opportunities before and after school to complement the school’s regular academic program. It features partnerships with school districts and community-based organizations to programs target students from high-poverty neighborhoods and attending low-performing schools.

A&O’s evaluation activities include systematic observations at all sites, interviews with staff and administrators, surveys of all stakeholder (students, staff and parents) and extensive analysis of student academic data. A&O has worked with large multi-site school districts as well as smaller community agency sites, including The Syracuse City School District, Watervliet Central School District, The YMCA of Central New York, and the Center for Community Alternatives.

Program Directors consistently note the value of these rigorous evaluations and the ongoing feedback and technical assistance for program improvement that the A&O team provides. These independent evaluations have provided the needed evidence and critical information to convince legislators at all levels of government to continue funding these after school programs.

21st Century Learning Communities

A&O conducts local evaluations for 21st Century Community Learning Centers (21st CCLC), a federally-funded program supporting high quality learning centers for Kindergarten through Grade 12 students. It features partnerships with school districts and community based youth development agencies.  The programs target students who attend high-poverty and low-performing schools. Tutoring, academic enrichment and a broad array of youth development opportunities are offered to complement the school’s regular academic program.

In addition, literacy and other educational services are offered to families of students participating in the program. Evaluation activities include systematic observations at all sites, analysis of student academic data provided by the school district as well as surveys completed by the students themselves. A&O has worked with large multi-site school districts as well as smaller community agency sites.

Unity House is a large and complex human service agency. Their mission is to empower people in recovery from substance abuse, coping with mental illness or diagnosed with a developmental disability.  Apter & O’Connor’s work with Unity House started with a strategic planning process. The resulting three-year plan guided their preparation for NY State’s service-delivery and reimbursement transformation resulting from Managed Care. The process explored new business opportunities, planned for updating all administrative systems for new compliance requirement, designed a robust performance management (PM) infrastructure to support high quality programming and addressed the steps needed to strengthen their marketing and community engagement.

Following the strategic planning process, A&O was engaged to provide professional development and lead a newly convened PM committee. Over the course of 18 months, the committee examined each program area to articulate intended outcomes, identify performance measures, collect and analyze data, develop user-friendly reports and dashboards, and plan for program adjustments and improvements. The work with A&O concluded but the PM Committee continues with data-driven-decision-making now a part of the agency culture.

Huntington Family Centers

The Huntington Family Centers Brighton Tower’s Service Coordination Program needed an independent quality assurance review on a tight timeline. Funded by the U.S. Department of Housing and Urban Development, the program provides residents in this 600 unit subsidized senior housing complex with the personal assistance they need to access resources and services.

The review examined the 12-month period ending June 30, 2014. A&O worked with residents, apartment building staff and Huntington to answer the three questions all funders and providers need answered: 1) Did we implement with fidelity? 2) Did we do it with quality? 3) Did our services have an impact on our constituents? We conducted interviews, held group discussions with residents, and used questionnaires to collect anonymous feedback. Sixty residents participated (approximately 10% of total residents, and 25% of the residents served in 2013). We wanted to know about their satisfaction and utilization of the services offered as well as barriers the residents faced in accessing them. We also gathered ideas for improvement for the Service Coordination Program.

In addition to providing the deliverable of a high quality independent program evaluation for the client, our process of communicating with residents about the Service Coordination Program increased their understanding of this valuable resource.

High School Graduation Initiative

SCSD was awarded a four year Federal grant to tackle high dropout rates and low graduation rates.  The District has contracted with Apter & O’Connor to evaluate the program and document implementation successes and challenges. The High School Graduation Initiative (HSGI) supports any student currently out of school, at-risk of dropping out or not on track for graduation. HSGI adds a strong focus on school attendance with the formation of Attendance Teams to track students early on and assist those with attendance issues through additional supports or alternative programming.  Another important component are the Graduation Initiative Specialists (or Case Mangers) who work directly with students and their families. A&O is conducting a formative and outcome evaluation, utilizing data from the Basic Educational Data System (e-School), standardized test scores, attendance records, and disciplinary referral reports as well as survey/interview with students, counselors and teachers and administrators to examine success of implementation and impact of the interventions. The findings and recommendations will inform further decision making for student supports and improved achievement.

Early Reading First

Early Reading First (ERF) was an early literacy program targeting preschoolers from low income families at high risk of reading failure. Funded by the U.S. Department of Education, each of these projects was a multi-year contract (3-6 years) with features unique to each district. Apter & O’Connor provided rigorous evaluations to support continuous program improvement and to assess both fidelity of implementation and the impact of the ERF interventions. Each scope of work required a detailed evaluation plan, extensive data collection and analysis, and extensive and detailed reporting to satisfy all US DOE GPRA requirements.  Our evaluation outcomes enabled the districts to design and sustain changes in professional development practices and classroom instruction, gain a place on the federal What Works Clearinghouse for early childhood best practices, initiate further research on critical factors for student gains, and leverage their data to support applications for additional grant funding.

Cities First Readers (CFR) is a collaboration of numerous nonprofits and all three public library systems across New York City designed to support the development of a literacy rich ecosystem. Their overarching goal is to ensure young children and their families receive needed supports to become successful readers. The collaborative partners are implementing numerous services and programs including parent education, book distributions, guided reading programs, professional development, home visiting and public awareness campaigns. A&O led the partners through a process of clarifying their goals, outcomes and strategies as well as each partner’s capacity for evaluation and their role within the collaborative. They were then ready to take the next step and document the benefits or impacts of this extensive endeavor.   A&O conducts several independent evaluation inquiries each year utilizing a variety of methods analysis for statistical significance using standardized assessments, geocoding and mapping, descriptive illustrations of service statistics, interviews, observations, and surveys of numerous stakeholder groups. A few of these recent and current inquires have examined:

• the strength and maturity of the coalition using the lens of Collective Impact[1],
• whether targeted CFR interventions lead to improvements in knowledge, skill, and behavior among parents to support early literacy for young children
• the concentration of CFR services as related to poverty levels across all NY City council districts
• if attendance is improving at library story times within high-risk neighborhoods? Is there a correlation with the CFR public awareness campaign?
• how has CFR adapted and responded in the time of the COVID19 pandemic
• is there consistency across CFR partners in the use of evidence-based/ best practices for book distribution and associated educational messaging to support home environments and early literacy development?

A&O was chosen for their years of evaluation experience in early childhood and literacy related programming and is thrilled to support this important initiative. 

[1] Preskill, H., Parkhurst, Marcie, Juster, J.S., Guide to Evaluating Collective Impact; Collective Impact Forum, FSG.

21st Century Community Learning Centers (21st CCLC) and the Extended Day/School Violence Prevention (ED/SVP) programs

A&O conducts evaluations for several 21st Century Community Learning Centers (21st CCLC) and the Extended Day/School Violence Prevention (ED/SVP) programs. These programs are federally and state funded to support tutoring, academic enrichment and a broad array of youth development opportunities before and after school to complement the school’s regular academic program. It features partnerships with school districts and community-based organizations to programs target students from high-poverty neighborhoods and attending low-performing schools.

A&O’s evaluation activities include systematic observations at all sites, interviews with staff and administrators, surveys of all stakeholder (students, staff and parents) and extensive analysis of student academic data. A&O has worked with large multi-site school districts as well as smaller community agency sites, including The Syracuse City School District, Watervlieit Central School District, The YMCA of Central New York, and the Center for Community Alternatives.

Program Directors consistently note the value of these rigorous evaluations and the ongoing feedback and technical assistance for program improvement that the A&O team provides. These independent evaluations have provided the needed evidence and critical information to convince legislators at all levels of government to continue funding these after school programs.

A&O Senior Associate Vicki Quigley has received high praise for her work with The Erie Canal Museum, Canal Landing in Chittenango, and The Skaneateles Historical Society, all located in Central New York.

One client told us, “This is one invoice we are happy to pay. I don't have enough words to thank Vicki for all her help and good work. We could not have achieved all we have without her.” 

Many arts organizations, struggling post-pandemic, are seeing the benefit of revisiting and, in some cases, re-envisioning their mission and activities. To the tough and rewarding strategic planning process, Vicki brings her experience in not-for-profit and museum management. Her legal knowledge (she holds a J.D. from Syracuse University College of Law, where she specialized in issues of charitable corporations) is an additional benefit to our clients. 

The Gifford Foundation: ADVANS Consultation

Apter & O’Connor provides consultation and assistance to a wide array of not-for-profit agencies to increase their ability to respond and thrive in challenging fiscal climates. Our capacity-building work with health, educational and human service agencies is taking several forms, depending on the needs and “life-cycle” stage of the organization. As part of this renewed emphasis on capacity building and organizational development, A&O President, Cynthia O’Connor, has served as a consultant for ADVANS (Advancing and Developing the Value & Assets of Non-Profits in Syracuse), an initiative developed and funded by the Gifford Foundation and the Ready or Not initiative supported by the Health Foundation of Western and Central New York (HFWCNY).

Cynthia was certified by Dr. Susan Kenny Stevens, Gifford’s lead consultant for the project and the author of the award-winning book Nonprofit Lifecycles Stage–Based Wisdom for Nonprofit Capacity. This organizational development framework assumes predictable needs, challenges, and strengths dependent on an agency’s life cycle — from start up to maturity — and works intensively with agency leadership teams to assess and fortify their infrastructure across management, governance, financial, and administrative systems, as well as programs and services.

Cynthia has had the privilege of three 2-year engagements through the Gifford ADVANS program and a fourth for the HFWCNY’s Ready or Not initiative.

Literacy Powerline

Apter & O’Connor is collaborating with the national organization, Literacy Powerline. The mission of Literacy Powerline is to increase literacy levels through effective and sustainable community collaboration and engagement. Literacy Powerline works with educators, business and labor leaders, philanthropic organizations, civic and faith-based groups, policymakers, literacy providers and students to demonstrate that everyone thrives when a community is committed to 100% literacy.  Scores of communities across the country have accessed the resources and capabilities of the Literacy Powerline team.  A&O is offering strategic planning, needs assessment and evaluation expertise to coalitions across the country to develop effective and highly accountable literacy initiatives within their own communities.  We are excited to be part of the Literacy Powerline team working with the Annie E. Casey National Support Center for the Campaign for Grade Level Reading to provide consultation for accountability systems, implementation and sustainability efforts of funded projects, particularly in rural areas.

The Campaign for Grade-Level Reading

Closer to home, Apter & O’Connor has designed an evaluation of the Herkimer Oneida Literacy Coalition for Literacy Powerline.  The approach is an empowerment model to facilitate stakeholder ownership of the evaluation activities and allow for on-going feedback, dialogue and appropriate adjustment.  The evaluation is designed to measure the development of a fully functioning literacy coalition and the impact on the community.

A&O has joined Literacy Powerline to support the Campaign for Grade-Level Reading in communities across New York State. The Campaign is a collaborative effort by foundations, nonprofit partners, states and communities across the nation to ensure that more children in low-income families succeed in school and graduate prepared for college, a career, and active citizenship. The Campaign focuses on an important predictor of school success and high school graduation—grade-level reading by the end of third grade.  Recognizing that schools can’t “do it alone”, the Campaign seeks to engage and mobilize whole communities to remove barriers, expand learning opportunities and assist parents to serve as partners in their child’s education.  A&O along with Literacy Powerline is helping to build a statewide network of GLR Communities across New York by offering coalition building, coaching, technical assistance and resources to individual communities and cities, so they can build their own unique networks and realize the goals of the Campaign.