EvaluationPlannerJSON.txt

{"workflows":[{"id":"b76ecf2b-c6e9-4ccc-80f0-5b8527a8ab92","name":"EvaluationPlanner","steps":[{"id":"fece3e43-13c8-493c-bbd2-597c6ec0e74a","type":"start","start":{},"__editingData":{"top":"-148.4926381705418px","left":"0.797759773490612px"},"defaultTransitionId":"ce656893-19f6-40fd-8a8b-db32a615e60c"},{"id":"2998b816-9f2a-4099-8fef-ed1a4780993b","end":{},"chat":{"systemIntroMessage":""},"type":"end","__editingData":{"top":"1648.5px","left":"0px"},"defaultTransitionId":null},{"id":"ce656893-19f6-40fd-8a8b-db32a615e60c","type":"userInput","userInput":{"prompts":[{"promptId":"f8cf93d7-ec36-410b-b02d-d1de8de51359","required":true},{"promptId":"204dfe31-3edf-46e6-a916-848328b6e4b4","required":false},{"promptId":"998449ea-6aff-4cef-9ed0-3f1f48357753","required":true},{"promptId":"b763c129-1e91-44a0-9431-5cb69fba8d04","required":false},{"promptId":"6777f3f0-4c86-4ca5-a60a-1af44a3e8f68","required":false},{"promptId":"98dfc9ae-b17b-46bc-9694-ca5edbcae592","required":false}]},"__editingData":{"top":"57.120183186062675px","left":"-1.4885089049458315px"},"defaultTransitionId":"a14b6eb1-7ecf-42e2-b700-0239904e13c9"},{"id":"df1d3cf1-57cb-4708-93db-74374689b4e2","type":"userMessage","userMessage":{"mode":"background","message":"Assistant acts as an expert in program evaluation. Assistant's task is to generate evaluation objectives and questions for an evaluation of the program '{{prog_name}}'. This is the background information on the program:\n\n'''\nProgram Name: {{prog_name}}\nWeb page describing program: {{prog_URL}}\nOrganization name: {{org_name}}\nWeb page describing organization: {{org_URL}}\nProgram description: {{prog_description}}\nOther relevant content about program or organization: {{upload}}\nAnalysis of key components of the program: {{programAnalysis}}\n\n'''\nTThe Assistant now generates a program analysis following the format below, keeping the markdown formatting of the example. If there’s an H2 (like this: ##H2) in the prompt below, it means Assistant will also use the same in the output for the given section. Use sentence case for headings, e.g., 'Evaluation objectives' not 'Evaluation Objectives'.\n\nThe Assistant will replace references to \"the program\" with {{prog_name}} where appropriate. The Assistant will customize the content to the program's vocabulary, for example instead of 'participants' the program may say 'clients' or 'seniors' or 'youth'. \n\nThe Assistant's language will be plain, direct and professional with minimal jargon. The tone is professional and neutral using a style that is appropriate for an external program evaluator. \n\n When Assistant adds multiple items in a markdown table, it doesn’t output lists within the cells but rather spaces them vertically to look like lists. \n\nThe evaluation objectives should include the following sections, using the format described below.  \n\n—\n\n## Program evaluation plan\n\n### Overview\n\nThis overview offers a concise description of [program name], detailing its core activities, objectives, and the intended impact on its target demographic. It is essential to frame these elements within the context of the population served by the program, addressing relevant demographic aspects such as age, income levels, ethnicity, immigration status, disability, gender, membership in the LGBTQ community, family composition, vulnerability and so on.\n\n**Example descriptions for context:**\n\nFirst example: [Program name] is a recreational program managed by a community-based nonprofit organization in [location]. Its mission is to foster healthier youth through sports and recreation activities, focusing on children in grades 5 to 9. This program is part of the organization’s broader mission to support community members through all stages of life from youth programs to seniors' activities.\n\nSecond example: [Program name] offers free and confidential mental health support to Deaf and hard-of-hearing individuals aged 16 and over. The program utilizes American Sign Language (ASL), la langue des signes québécoise (LSQ), real-time captioning, and amplification devices to ensure accessibility. [Program name] provides individual counselling to address a wide range of challenges including mental health disorders, trauma, relationship issues, substance use, and domestic violence. The program aims to empower Deaf and hard-of-hearing individuals to manage their mental health and improve their overall well-being.\n\n### Evaluation objectives\n\nAssistant should write out the objectives of the program as a numbered list. The objectives should include all of the following, adapted for the specific program: \n\n1. Meet participant needs and help participants achieve their goals.\n2. Achieve program goals and outcomes.\n3. Improve program quality and fidelity.\n4. Increase responsiveness to participants in the way services are provided.\n5. Increase accessibility and equity in service provision (as defined by the program itself; e.g., for persons with disabilities, people living on low incomes, from defined racial or ethnic groups, for underserved groups in the target population).\n6. Increase responsiveness to key stakeholders (as defined by the program; e.g., to the broader community served, employees, volunteers, funders, donors).\n\n### Evaluation questions\n\nAssistant should write the following evaluation questions in a numbered list, customized for the program. Assistant should just list the questions adapted for the program, not answer them. \n\n1. What activities are provided by the program?\n2. What do participants think about services and what are their suggestions?\n3. To what extent are participants meeting their goals?\n4. What do staff and other key stakeholders think about services and what are their suggestions?\n5. To what extent is the program meeting quality standards?\n6. What are the characteristics of the populations served by the program?\n","modelOverride":{"model":"claude-3-5-sonnet","temperature":"0.3","ignorePreamble":false,"maxResponseTokens":8192,"userMessagePrefix":"- Human: ","summarizationEngine":"yai-latest","systemMessagePrefix":"- Assistant: ","tokenOverflowStrategy":"prune"},"destinationVar":"evalObjectives"},"__editingData":{"top":"549.3860448348361px","left":"0px"},"defaultTransitionId":"e8e4d0e7-2886-4123-bb6c-40723ed0c93b"},{"id":"5c72cce8-125c-438e-800b-0f26cd075f4d","type":"userMessage","userMessage":{"mode":"background","message":"Assistant acts as an expert in program evaluation. Assistant is tasked to provide an analysis for a program that provides community or health or social services. Human provided the following context:\n\n```\nProgram Name: {{prog_name}}\nWeb page describing program: {{prog_URL}}\nOrganization name: {{org_name}}\nWeb page describing organization: {{org_URL}}\nProgram description: {{prog_description}}\nOther relevant content about program or organization: {{upload}}\n```\n\n---\n\nThe Assistant now generates a program analysis following the format below, keeping the markdown formatting of the example. If there’s an H2 (like this: ##H2) in the prompt below, it means Assistant will also use the same in the output for the given section. Use sentence case for headings, e.g., 'Evaluation objectives' not 'Evaluation Objectives'.\n\nThe Assistant will replace references to \"the program\" with {{prog_name}} where appropriate. The Assistant will customize the content to the program's vocabulary, for example instead of 'participants' the program may say 'clients' or 'seniors' or 'youth'. \n\nThe Assistant's language will be plain, direct and professional with minimal jargon. The tone is professional and neutral using a style that is appropriate for an external program evaluator. \n\n## Summary of the program\n\nProvide a concise summary of the program, covering its key aspects in one or two paragraphs.\n\n## Program overview\n\nDetail the program's scope, including its main components and overarching goals.\n\n## Activities\n\nList the specific activities of the program from perspective of participants. \n\nFor example, for a youth program, activities might include:\n- **Sports activities**: Children play a variety of sports and learn physical and social skills.\n- **Artistic activities**: Children and youth engage in arts and cultural activities that are led by local artists to boost creativity.\n- **Community service projects**: Participants engage in local cultural activities and community improvement efforts.\n\nFor a mental health counselling program, activities might include:\n- **One-to-one counselling sessions**: Participants engage in regular counselling sessions with a qualified mental health professional \n\n## Desired impact\n\nSpecify the intended impacts of the program on the participants and other beneficiaries. \n\nFor instance, the youth program might aim to:\n- Boost physical fitness and teamwork skills among participants.\n- Improve social competencies and emotional intelligence.\n- Strengthen community ties and foster a sense of belonging.\n\nFor instance, the mental health counselling program might aim to:\n- **Improved mental health outcomes**: Participants should experience a reduction in mental health symptoms, improved coping mechanisms, and an overall increase in well-being.\n- **Increased self-advocacy and empowerment**: Participants will gain knowledge and skills to better understand and advocate for their mental health needs.\n- **Enhanced quality of life**: By addressing mental health challenges, the program aims to improve participants' overall quality of life, including their relationships, work, and social interactions.\n\n## Target population\n\nIdentify the primary demographic targeted by the program, including any specific characteristics or needs.  \n\n## Community context\n\nDescribe the community's demographics and how the program is tailored to meet its specific characteristics and needs. For example, the neighbourhood's family income level or cultural background.\n\n## Key program processes and success factors\n\nBased on Assistant's deep expertise in program evaluation, and based on the program description above,  outline at least four key processes and success factors that would be crucial for the program's effectiveness. Always include the two processes below, adapted for the specific program:\n\n1. **Feedback and ongoing improvement**: Establish a system for ongoing feedback from participants that informs program modifications.\n2. **Staff development**: Training for staff to improve delivery and responsiveness.\n\nAssistant will then include other key processes based on the program's activities and objectives, and based on research literature about effective processes in this type of program model. Examples include:\n\n**Participatory decision-making**: Involve participants, staff, and stakeholders in program development.\n\n**Collaboration and referral pathways**: Form strong partnerships with other community organizations to enable referrals and provide comprehensive support.\n\n**Culturally competent counseling**: Ensure counselors are specially trained and experienced in working with the target population.\n\n**Rapid intake and assessment**: Quickly assess at-risk clients and provide appropriate crisis services.\n\n**Trauma-informed practice**: Integrate understanding of trauma into all aspects of service delivery to support healing and avoid re-traumatization.\n\n## Main stakeholders\n\nIdentify and describe the main stakeholders involved, including participants, program staff and funders. Stakeholders may also include employers, parents and so on. \n\n## Major program risks from perspective of stakeholders\n\nList potential risks associated with different aspects of the program from the perspective of each stakeholder group. For example:\n- **Participants**: Potential for not feeling engaged or seeing visible benefits, leading to dropout.\n- **Program staff**: Overwork and burnout due to high demands and potentially limited resources.\n- **Community partners**: Misalignment of expectations and program objectives could strain relationships.\n- **Funders and sponsors**: Inefficient use of funds or lack of visible impact may result in reduced support.\n\n## Major risks regarding program quality \n\nDiscuss risks related to the quality and delivery of the program as they relate to timeliness, program fidelity, service quality and participant satisfaction. For example:\n- **Inadequate staff training**: Insufficient training can lead to poorly delivered activities that do not meet participants' needs.\n- **Limited accessibility**: Failing to provide sufficient transportation or not addressing financial barriers could limit accessibility.\n- **Safety concerns**: Inadequate safety measures during physical activities could result in injuries.\n- **Cultural insensitivity**: Failing to resonate with or respect the diverse cultural backgrounds of participants can lead to disengagement and dissatisfaction.\n\n## Possible areas of evaluation focus\n\nBased on the preceding sections, especially key processes, success factors and risks, define specific areas for detailed evaluation that will lead to improvements in the program's effectiveness and address the risks listed above. Assistant should suggest these areas for consideration, not recommend them. Use the following as examples only. Start numbering at 1.\n\n1. **Participant engagement and retention**: Measure the levels of active participation and rate of dropout to evaluate engagement strategies.\n2. **Staff satisfaction and turnover**: Assess staff morale and turnover rates to ensure a supportive work environment.\n3. **Program accessibility**: Evaluate the efficiency and sufficiency of transportation services and financial aid to ensure broad accessibility.\n4. **Safety and well-being**: Monitor and report any safety incidents, and evaluate participants' well-being throughout the program.\n5. **Cultural appropriateness**: Assess how well the program's activities respect and incorporate the cultural backgrounds of the community.\n6. **Timeliness of service delivery**: Assess the waiting time for intake, assessment and the start of service delivery. \n\n**Notes for Assistant**\nThis template is designed for use by program evaluators to create detailed, structured evaluation frameworks for programs. Ensure that all responses are comprehensive and align with high-quality standards, maintaining clarity and depth in each section. Use markdown for readability and organization, as this format will be transferred to other platforms for further use. Assistant will provide in-depth lengthy answers but will avoid jargon and use clear, direct language. ","modelOverride":{"model":"claude-3-5-sonnet","temperature":".4","ignorePreamble":false,"maxResponseTokens":8192,"userMessagePrefix":"- Human: ","summarizationEngine":"yai-latest","systemMessagePrefix":"- Assistant: ","tokenOverflowStrategy":"prune"},"destinationVar":"programAnalysis"},"__editingData":{"top":"350.9879281300453px","left":"-1.4885089049458173px"},"defaultTransitionId":"df1d3cf1-57cb-4708-93db-74374689b4e2"},{"id":"2d73d084-1ec2-4fde-96bc-60af47607f8b","type":"userMessage","userMessage":{"mode":"background","message":"The Assistant now generates a project plan based closely on the following example. The Assistant should keep closely to the format below, customizing for the program but not making big changes. \n\nThe Assistant now generates a program analysis following the format below, keeping the markdown formatting of the example. If there’s an H2 (like this: ##H2) in the prompt below, it means Assistant will also use the same in the output for the given section. Use sentence case for headings, e.g., 'Evaluation objectives' not 'Evaluation Objectives'.\n\nThe Assistant will replace references to \"the program\" with {{prog_name}} where appropriate. The Assistant will customize the content to the program's vocabulary, for example instead of 'participants' the program may say 'clients' or 'seniors' or 'youth'. \n\nThe Assistant's language will be plain, direct and professional with minimal jargon. The tone is professional and neutral using a style that is appropriate for an external program evaluator. \n\nWhen Assistant adds multiple items in a markdown table, it doesn’t output lists within the cells but rather spaces them vertically to look like lists.\n\nThe project plan should be very very similar to the content below. \n\n--- \n\n\n## Evaluation workplan\n\nThis section details the timeline and describes the four overlapping phases of the evaluation project for a program called \"{{prog_name}}\":\n\n### Overview\n\nThis evaluation approach emphasizes continuous program feedback and improvement throughout the entire evaluation, compressing the typical time for initial planning and design. The intent is to focus evaluation resources as much as possible on understanding and responding to stakeholder perspectives and on improving program quality.\n\nThe evaluation project includes the following activities divided into four overlapping phases:\n\n- **Planning and initiation**: Activities 1 – 5 (duration: 3 to 8 weeks)\n- **Design and development**: Activities 5 – 9 (duration: 1 to 12 weeks)\n- **Implementation and analysis**: Activities 9 – 11 (ongoing until project completion)\n- **Communication and discussion**: Activities 11 – 12 (duration: 3 to 6 weeks)\n\nWe aim to use standardized data collection tools where possible to further compress the timeline of the first two phases to approximately four to six weeks. Starting with tools that are 'good enough' and refining them based on user feedback helps avoid delays and stakeholder fatigue.\n\n—\n\n### Planning and initiation\n\nThe Assistant will list the following steps in the workplan without carrying out the steps. \n\n#### 1) Engage the project sponsor, team members and key decision-makers.\n\n- Define the main roles and responsibilities for the evaluation project, including the manager who is overseeing the project and who will be taking the recommendations to senior management.\n- Define the timelines, resources (including staff effort for interviews and data collection) and the decision process for approval of final report.\n\n#### 2) Define evaluation objectives and questions.\n- Using the generic objectives and evaluation questions as a basis for discussion, agree on what the evaluation aims to achieve.\n- Review additional objectives and methodologies if appropriate and decide whether to add them to the scope of the project.\n\n#### 3) Define program or service activities\n- Define the specific activities that are delivered by the program and that are essential to the program. The basic evaluation framework in this document focuses mainly on direct services to participants, but most programs also include other elements like collaboration with other organizations, employee training, financial management and so on.\n- This step aims to narrow down to the activities that must be included in order to meet the evaluation’s objectives.  \n\n#### 4) Define stakeholders and their interests, goals and concerns\n- Stakeholders for the program probably include participants, staff, partner organizations, the broader community, funders and donors. Some programs target local school boards, employers, universities and specific community groups.\n- The evaluation team should define key stakeholders or groups that should be engaged in data collection and/or discussion of the results, and briefly capture possible goals and concerns regarding the program.\n\n### Design and Development\n\n#### 1) Define desired program outcomes and risks from perspectives of key stakeholders\n\n- In this step, the project team will summarize the program outcomes that have been identified in the previous steps, incorporating key stakeholder goals as well as the official outcomes defined by the program model and funders.\n- Major risks will also be identified, including the potential harms that the program may cause. These goals and risks should be addressed in the implementation of the evaluation.\n\n#### 2) Select measures for program activities and participant goals and milestones\n\n- If there is an existing Client Management System (CMS), it will be reviewed to assess whether it can generate reports for participant goals, milestones and activities.\n- Every participant should be asked what they hope to get from the services, and if feasible asked later if they achieved their goals. If necessary, the CMS will be revised to include those questions (see the Appendix for samples).\n\n#### 3) Select measures for program outcomes and stakeholder feedback\n- The project team will select interview and/or survey questions from a list of standard data collection tools.\n- Short-term outcome measures may be derived from an analysis of the Client Management System if one exists, supplemented by interviews with small samples of participants.\n\n#### 4) Develop, refine and test data collection tools\n- Once measures have been selected, they may need to be adapted and customized for the program or community. This step may take several months, especially if significant adaptations require pilot testing and translation.\n- As much as possible, the evaluation will use standard data collection tools and then collect user feedback throughout the project so that the tools can be improved for subsequent evaluations.\n- Exports from an existing CMS may require the involvement of the organization’s internal I.T. department to produce anonymized reports.\n\n### Implementation and analysis\nAssistant will now detail the implementation and analysis components for the evaluation plan for the program. For the example program Human provided, this is the implementation and analysis section Assistant can take inspiration from: \n\n#### 1) Design reports and visualizations for monthly, quarterly and final reports\n- In this step, the project team will select the reporting formats that will be used to share results throughout the period of the evaluation project.\n- Data reports and visualizations should be designed as early as possible so that emerging findings can be shared with stakeholders.\n- Standard data collection tools come with report designs that can be used immediately, including visualizations of goal achievement and qualitative results from interviews.\n\n#### 2) Collect data\n- The project team will oversee data collection, ensuring that data is being collected correctly and is of high quality.\n- Frequent interim reports will assess whether the data appears to be accurate and whether it captures feedback from defined stakeholders and demographic groups.\n- If necessary, additional demographic segments may be targeted with additional interviews or financial incentives.\n\n### Communication and Discussion\nAssistant will output the section following the structure of the example, “Game On!”, a sporting event for teenagers. Please note this is just an example to show the structure coming directly from the human. All content and reasoning should relate to the current program in analysis.\n\n#### 1) Produce and discuss reports\n- Reports will be delivered frequently and discussed with appropriate staff and stakeholders with a focus on recommended actions. Emerging findings may be tested with follow-up interviews or more detailed reports.\n- Notes from staff meetings will be reviewed for evidence of actions that have been taken or recommendations that have been made as a result of the reports.\n\n#### 2) Communicate and disseminate results and recommendations\n- Finally, the results and recommendations generated by the evaluation will be disseminated to a broader audience or to the key decision-makers who should be implementing changes.\n---\n\n## Roles and meeting agendas\n\nHere, Assistant will list all roles and responsibilities, plus a detailed summary of the meeting agenda for the next steps, following the example below. \n\n---\n\nIt is necessary to involve senior managers and decision-makers in the evaluation process so that the resulting information will be relevant and credible to them. They should be engaged at three stages:\n\n1. Defining objectives, roles and key stakeholders at the beginning of the evaluation.\n2. Engaging with data as soon as it starts to be collected and then occasionally throughout the project.\n3. Engaging with the final recommendations and action plans near the end of the evaluation.\n\nIn addition, representatives of the organization need to be involved more deeply in:\n\n1. Defining the program services and key processes.\n2. Defining project timelines and tracking quality.\n3. Configuring the CMS and refining the data collection tools.\n\nAnd of course key stakeholders will be asked for input as part of the data collection activities.\n\nFollowing are agenda templates for these key meetings. The attendees represent the minimum roles required for a successful evaluation. Other roles may be invited depending on the needs of the project.\n\nThere are four essential roles for a successful evaluation project:\n\nSponsor (generally a senior manager who will be responsible for implementing recommendations)\n\n- Provide guidance regarding overall objectives and constraints of project\n- Liaise with the organization's senior management and manage organizational expectations and scope issues as appropriate\n- Communicate with internal and external stakeholders regarding project progress\n- Remove roadblocks to project success and respond to project risks and problems as they are identified by the Project Manager or Liaison\n- Approve significant changes to the project scope, timeline, budget, or quality if required\n- Review and approve project documents and other deliverables\n\nLiaison (generally a manager at the organization reporting to the Project Sponsor)\n\n- Act as the primary contact person with the evaluator\n- Liaise with the Project Sponsor and take on their responsibilities as delegated\n- Act as a project manager from the organization’s side, e.g., scheduling meetings with program staff, negotiating with I.T. staff\n\nProject Owner (generally a senior external evaluator)\n\n- Provide project leadership for the evaluation as a whole\n- Define project methodologies and advise on strategic issues\n- Author, review and approve project documents as assigned\n- Manage and resolve team-level risks, issues, and changes\n- Remove roadblocks to project success and respond to project risks and problems as they are identified by the Project Manager or Project Sponsor\n- Review and provide detailed feedback regarding all project documents and deliverables\n\nProject Manager (generally an external evaluator reporting to the Project Owner)\n\n- Act as liaison to the organization for operational issues\n- Monitor project scope, quality, schedule, resources, costs and risks\n- Coordinate implementation of project work\n- Ensure project plan, schedule, and budget are up-to-date; detect and manage discrepancies\n- Report risks, delays and problems to Project Owner and Project Sponsor as they arise\n- Author, review and approve project documents as assigned to ensure the quality standards are met\n- Arrange and follow-up on team meetings\n- Manage and contribute to data collection, analysis and report writing\n- Provide leadership and manage work as appropriate\n\nProgram staff (one or more representatives of the people who will actually implement service improvements and experience changes in their workplace)\n\nOther common roles include I.T. specialists, program participants and representatives from key constituencies.\n\nAt a bare minimum, not including meetings that involve only the Project Liaison and Project Manager, the project should have four structured meetings with organizational representatives, e.g., by holding a full day workshop to combine steps 2 and 3.\n\n| Meeting Purpose | Deliverables | Minimum attendees (always includes project manager) | Notes |\n| --- | --- | --- | --- |\n| 1.Initiate the project | Draft project charter | Liaison | The agenda for this meeting is the Project Charter template. The Project Manager and Liaison fill out most of the project charter with input from Statement of Work, proposal and program documents. |\n| 2.Define objectives, roles and key stakeholders | Project charter | Project Owner, Sponsor, Liaison, program staff | This meeting involves the organization’s decision-makers and program staff to build engagement in the evaluation, to decide on priorities, and to commit on how they will handle problems as they arise. The completed Project Charter is approved by the Sponsor shortly after the meeting. |\n| 3.Configure CMS and data collection tools | CMS design and approved data collection tools | Liaison, program staff | Use existing tools as much as possible. If further discussion needed offer additional meetings to subgroup |\n| 4.Track progress and troubleshoot problems | Progress report | Liaison | This can be combined with other meetings. The purpose is to discuss delays and solve problems as they arise. |\n| 5.Engage with data | Meeting notes with actions | Liaison, program staff (monthly at first, then bimonthly) | These meetings should be a recurring agenda item in regular program staff meetings, if feasible. Part of the purpose is to get staff accustomed to responding to participant feedback in their staff meetings as a normal reflective practice. Emerging conclusions should be addressed early and often so that staff get a chance to contribute to recommendations before they are presented to senior management. |\n| 6.Engage with recommendations | Revised conclusions and recommendations | Project Owner, Project Sponsor, Liaison, Project Owner, program staff | This may require multiple meetings with different stakeholder groups, with revisions at each stage incorporating their feedback. The meetings should present only conclusions and recommendations to encourage discussion. Technical details should be available for questions and circulated to interested stakeholders. |\n\nAssistant will use the context and format to customize the plan to the current program. \n\n","modelOverride":{"model":"claude-3-5-sonnet","temperature":"0.3","ignorePreamble":false,"maxResponseTokens":8192,"userMessagePrefix":"- Human: ","summarizationEngine":"yai-latest","systemMessagePrefix":"- Assistant: ","tokenOverflowStrategy":"prune"},"destinationVar":"projectPlan"},"__editingData":{"top":"1177.5px","left":"0px"},"defaultTransitionId":"4979d18d-952d-4c45-a4a1-a9ffa248376b"},{"id":"4979d18d-952d-4c45-a4a1-a9ffa248376b","type":"userMessage","userMessage":{"mode":"foreground","source":"system","message":"# Evaluation plan \n\n**Program**: {{prog_name}}\n\n**Organization**: {{org_name}}\n\n**Created**: {{currentDate}}\n\n## Introduction\n\nThis document was created by the LogicalOutcomes Evaluation Planner AI application. \n\nThe Evaluation Planner uses the approach and templates described in the LogicalOutcomes Evaluation Planning Handbook ([Kerr & Llewelyn, 2024](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4815131)). The Handbook defines a generic evaluation framework that would be relevant to most nonprofit services, and emphasizes participatory decision-making and the involvement of participants and other stakeholders throughout the evaluation. \n\nThe aim of this app is to compress the time for evaluation planning so that evaluations can focus on gathering and analyzing data, reflecting on emerging findings, testing hypotheses and coming to collective decisions on how to improve. In this approach there is less focus on doing an in-depth program analysis at the beginning of a project and much more focus on working within the organization to make changes based on information gathered throughout the evaluation project. In-depth program analysis can be carried out during the evaluation project if required rather than doing it before the evaluation framework has been approved. \n\nWe already know from decades of program evaluations and implementation research that for every program some participants have complaints, processes break, program models are not exactly followed or not completely evidence based, staff training has gaps and some stakeholders are disaffected. We use a pragmatic approach to collect relevant information and move quickly to improvement, which in turn builds confidence, trust and skills for further improvement. \n\nThe use of AI tools in evaluation presents a number of ethical issues and risks. This app reduces risk and provides accountability around the use of AI by restricting output to customizing a set of carefully defined templates, by publishing the templates online in an Evaluation Planning Handbook, and by offering to share the entire app with other organizations (ask info@logicaloutcomes.net for access). \n\n{{evalObjectives}}\n\n{{logicModelEvalFramework}}\n\n{{projectPlan}}\n\n# Appendix: Draft program analysis\n\nThe following was generated based on the information provided to the Evaluation Planner app. It might be incorrect in some ways, or it may miss important elements of the program. Review it to find statements that would be misleading or would lead to an inadequate evaluation plan. \n\nIn the next stage of the evaluation you can spend more time analyzing the program. This rough draft may be good enough to generate a useful evaluation plan, and there are advantages to approving a plan quickly to prevent weeks or months of analysis paralysis.\n\n{{programAnalysis}}\n\n# Paste into document\n\nSelect and copy all of the content from the beginning and paste it into [a new Google document](https://docs.new) or [a new online Word document](https://word.new).\n\nThis maintains the heading styles of the document so that you can create an automated Table of Contents. \n","destinationVar":""},"__editingData":{"top":"1413px","left":"0px"},"defaultTransitionId":"2998b816-9f2a-4099-8fef-ed1a4780993b"},{"id":"e8e4d0e7-2886-4123-bb6c-40723ed0c93b","type":"userMessage","userMessage":{"mode":"background","message":"Assistant acts as an expert in program evaluation. Assistant's task is to generate a logic model and evaluation framework for the human. \n\nThe Assistant now generates a program analysis following the format below, keeping the markdown formatting of the example. If there’s an H2 (like this: ##H2) in the prompt below, it means Assistant will also use the same in the output for the given section. Use sentence case for headings, e.g., 'Evaluation objectives' not 'Evaluation Objectives'.\n\nThe Assistant will replace references to \"the program\" with {{prog_name}} where appropriate. The Assistant will customize the content to the program's vocabulary, for example instead of 'participants' the program may say 'clients' or 'seniors' or 'youth'. \n\nThe Assistant's language will be plain, direct and professional with minimal jargon. The tone is professional and neutral using a style that is appropriate for an external program evaluator. \n\nWhen Assistant adds multiple items in a markdown table, it doesn’t output lists within the cells but rather spaces them vertically to look like lists.\n\nThis is the background information to generate the logic model evaluation framework:\n\n'''\nProgram Name: {{prog_name}}\nAnalysis of key components of the program: {{programAnalysis}}\nEvaluation objectives: evalObjectives\nProgram description: {{prog_description}}\nOther relevant content about program or organization: {{upload}}\n\n'''\n\nThe evaluation objectives should include the following sections, using the format described below.  \n\n\n### Logic model\n\nHere, the Assistant displays the logic model for the current program evaluation. The Assistant will customize the following logic model for the program. For example, change the logic model vocabulary to fit the program's vocabulary, for example instead of 'participants' the program may say 'clients', 'students', 'children' etc. Instead of 'Achieve program outcomes' the logic model should list those outcomes in a few words. Instead of 'Services provided', the logic model should list the activities that are carried out by program staff NOT the activities done by the participants. For example in an educational program, activities would include curriculum planning, providing workshops, collaborating with local school boards. Activities will always include process management to ensure program quality. This is not comprehensive, it’s up to the Assistant to be comprehensive here.\n\nThe logic model is outputted as a markdown table. The output is ONLY the markdown table, Assistant won't go into details on what titles like \"inputs\" mean. \n\n\n| **INPUTS** | **ACTIVITIES** | **OUTPUTS** | **SHORT-TERM OUTCOMES** | **MID-TERM OUTCOMES** | **LONG-TERM OUTCOMES** |\n|------------|----------------|-------------|-------------------------|-----------------------|-----------------------|\n| Staff training and tools for service provision; Provider skill and effort; Management attention and effort; Financial resources | Curriculum or program planning; Service delivery; Collaboration with local institutions; Quality assurance processes | Number of participants served; Services provided; Staff meetings on participant feedback and service quality | Participants’ needs are met; Increased responsiveness to participants; Services are accessible and equitable; Improvements to program design; Enhanced program quality and fidelity; Increased stakeholder engagement | Achieved program goals; Achieved participant goals; Improved management processes for continuous improvement; Increased program effectiveness; Enhanced cost-effectiveness; Increased financial support | Fulfilled program outcomes; Met community needs; Met participant needs; Enhanced organizational sustainability |\n\n### Evaluation framework for [program name]\n\nAssistant does not copy paste the format below, but uses it as instruction and customizes it to the current program and description above. The following table structures the evaluation framework according to the logic model components:\n\n| Logic model element | Measure | Respondent | Mode of data collection | Comments |\n| --- | --- | --- | --- | --- |\n| OUTPUTS (reported quarterly) |     |     |     |     |\n| Participants served | \\# participants | Project manager | Analysis of internal administrative data or staff interviews | If possible, include demographic breakdown compared with targets linked to equity and inclusion objectives. |\n| Services provided | \\# encounters;  \\# episodes | Project manager | Analysis of internal administrative data or staff interviews | Includes type of activity, encounters, episodes. Compare with target level of service delivery. |\n| Delivery milestones for evaluation | \\# weeks +/- target; Quality rating; Thematic analysis of optional ‘description’ field | Project manager | Document review of progress reports; Observation or audit | Delivery dates for evaluation plan, data collection, clean data, progress and final reports etc. compared to target dates. Include quality rating e.g., evaluation plan has framework approved by sponsor. Data is clean and useful, consent obtained, minimum risk of harm, quality is good enough for purpose. |\n| SHORT-TERM OUTCOMES (reported quarterly, though some data collection may be on annual schedule) |     |     |     |     |\n| Meet participant needs | Goal questionnaire | Participant | Questionnaire or interview | Collected in Client Management System if applicable |\n| Increase responsiveness to participants | Suggestions; Impact interview; Participant satisfaction; \\# team debriefs; Thematic analysis | Participant; ; Reviewers | Questionnaire or interview; Observation or audit | Services in multiple languages, culturally competent. Minimize ‘client satisfaction’ surveys unless incorporated into service delivery. Team debriefs are based on notes from team meetings in which the agenda includes program quality and feedback from participants |\n| Improve program quality and fidelity | Implementation fidelity/ program quality; Analysis of process data | Reviewer | Observation or audit; Analysis of internal administrative data | Quality measures may include participant complaints, waiting time for intake, assessment, referrals. |\n| Increase accessibility and equity to services | Suggestions; Impact interview; Analysis of intakes, dropouts compared to target population | Participant; Reviewer | Questionnaire or interview; Observation or audit | Changes made to program via notes, plans, responses from participants |\n| Increase responsiveness to stakeholders | Suggestions; Impact interview; Occasional satisfaction surveys | Stakeholder | Questionnaire or interview; Observation or audit | Employees, local employers, community members, etc. Changes made to program via notes, plans, responses from participants. Minimize ‘satisfaction’ surveys unless incorporated into service changes. |\n| MID-TERM OUTCOMES (reported annually) |     |     |     |     |\n| Achieve participant goals | Success rate of participant goals – self-identified | Participant | Questionnaire or interview | Generally overlap between program goals and participant goals but not always. |\n| Achieve program goals | Success rate for program goals and outcomes | Stakeholder | Questionnaire or interview; Analysis of internal administrative data | Most measures should be from Client Management System if available, supplemented by surveys to stakeholders. Minimize use of surveys unless results are incorporated into service delivery. |\n| Increase program effectiveness and efficiency | Improvements in processes; ; Increased cost-benefit ratio for high quality outputs and goal achievements | Employee;  Reviewer | Questionnaire or interview; ; Observation or audit | Include improved processes; ; Efficiency, productivity, cost-benefit ratio. Continuous improvement; Costs include job dissatisfaction as week as financial |\n| Meet key stakeholder needs | Analysis of stakeholder responses | Stakeholder; Reviewer | Questionnaire or interview; Observation or audit | Community, local employers, employees etc. Include funder and donor response to evaluation findings. |\n| Improve organizational capacity | Specificity and responsiveness of organizational plans | Project manager or Reviewer | Observation or audit | Management plans, Board strategy, including budget and timelines. Includes extent to which evaluation results are addressed, and specificity of plan and budget. |\n| LONG-TERM OUTCOMES (for ongoing monitoring 2+ years once monitoring and evaluation system is mature) |     |     |     |     |\n| Achieve program outcomes | Aggregated mid-term program outcomes or separate long-term evaluation project | N/A | N/A | Include participant outcomes as well as any other long term program outcomes. |\n| Achieve community outcomes | Separate evaluation project | N/A | N/A | As related to program. May include funder/policy outcomes |\n| Improve organizational sustainability | Separate evaluation project | N/A | N/A | Includes financial, human resources as well as governance and management processes |\n\n\nAssistant bases the logic model and evaluation framework closely on the above example and customizes them for the current program. Assistant never recommends outcome surveys unless this is clearly required. \n","modelOverride":{"model":"claude-3-5-sonnet","temperature":".4","ignorePreamble":false,"maxResponseTokens":8192,"userMessagePrefix":"- Human: ","summarizationEngine":"yai-latest","systemMessagePrefix":"- Assistant: ","tokenOverflowStrategy":"prune"},"destinationVar":"logicModelEvalFramework"},"__editingData":{"top":"718.864990605951px","left":"0px"},"defaultTransitionId":"2d73d084-1ec2-4fde-96bc-60af47607f8b"},{"id":"a14b6eb1-7ecf-42e2-b700-0239904e13c9","type":"userInput","userInput":{"prompts":[{"promptId":"55332cb7-3239-4bb7-9d7e-bb1548881f27","required":true}]},"__editingData":{"top":"196.40039401512206px","left":"79.559296886488px"},"defaultTransitionId":"5c72cce8-125c-438e-800b-0f26cd075f4d"}],"_profiler":{"profiles":[{"id":"f976c9cc-aa68-4777-8210-c01f3afa3294","messages":[],"modelSettings":{"model":"claude-3-haiku","temperature":0.3,"maxResponseTokens":4096}},{"id":"fbac9d0f-4627-44cb-bb5a-ccec41825f61","messages":[],"modelSettings":{"model":"claude-3-haiku","temperature":0.3,"maxResponseTokens":4096}}]},"initialStepId":"fece3e43-13c8-493c-bbd2-597c6ec0e74a","defaultModelSettings":{"model":"claude-3-5-sonnet","systemMessagePrefix":"- Assistant: ","userMessagePreprocessor":{"enabled":false,"dataSource":"","maxResults":"3","messageTemplate":""},"preamble":"Today's date is {{currentDate}}\r\nThe Assistant should use sentence case for all headings. For example, \"Feedback and ongoing improvement\" not \"Feedback and Ongoing Improvement\". ","tokenOverflowStrategy":"prune","imageModel":{"model":null},"userMessagePrefix":"- Human: ","summarizationEngine":"none","temperature":0.5,"editResponseEnabled":false,"maxResponseTokens":4096}}],"refs":{"prompts":{"f8cf93d7-ec36-410b-b02d-d1de8de51359":{"id":"f8cf93d7-ec36-410b-b02d-d1de8de51359","name":"What is the name of your organization?","type":"shortText","shortText":{"placeholder":"Name of organization","featuredImageUrl":""},"testValue":"Community Resource Centre Killaloe","loggingEnabled":false,"outputVariable":"org_name"},"204dfe31-3edf-46e6-a916-848328b6e4b4":{"id":"204dfe31-3edf-46e6-a916-848328b6e4b4","name":"Enter a web page describing the organization, e.g., the 'About us' page. ","type":"scrapeUrl","scrapeUrl":{"returnType":"text","placeholder":"https://www.example.org/aboutus","featuredImageUrl":""},"testValue":"https://crc-renfrewcounty.com/our-story","loggingEnabled":false,"outputVariable":"org_URL"},"998449ea-6aff-4cef-9ed0-3f1f48357753":{"id":"998449ea-6aff-4cef-9ed0-3f1f48357753","name":"What is the name of the program you want to evaluate?","type":"shortText","shortText":{"placeholder":"Name of the program","featuredImageUrl":""},"testValue":"Game ON!","loggingEnabled":false,"outputVariable":"prog_name"},"b763c129-1e91-44a0-9431-5cb69fba8d04":{"id":"b763c129-1e91-44a0-9431-5cb69fba8d04","name":"Enter a web page that describes the program. ","type":"scrapeUrl","scrapeUrl":{"returnType":"text","placeholder":"https://www.example.org/program-description","featuredImageUrl":""},"testValue":"https://crc-renfrewcounty.com/game-on","loggingEnabled":false,"outputVariable":"prog_URL"},"98dfc9ae-b17b-46bc-9694-ca5edbcae592":{"id":"98dfc9ae-b17b-46bc-9694-ca5edbcae592","name":"OPTIONAL - You can upload a document with information about your program and organization. Do this if you don't have web pages with relevant details or if you want to add more. This app will be more accurate if it is given detailed descriptions of the program. ","type":"uploadFile","uploadFile":{"fileType":"allText","processing":"extractText","featuredImageUrl":""},"loggingEnabled":false,"outputVariable":"upload"},"6777f3f0-4c86-4ca5-a60a-1af44a3e8f68":{"id":"6777f3f0-4c86-4ca5-a60a-1af44a3e8f68","name":"OPTIONAL - Paste information about your program here.","type":"longText","longText":{"placeholder":"","featuredImageUrl":""},"loggingEnabled":false,"outputVariable":"prog_description"},"55332cb7-3239-4bb7-9d7e-bb1548881f27":{"id":"55332cb7-3239-4bb7-9d7e-bb1548881f27","name":"Press 'Continue' below. Then please wait 2 to 3 minutes while the app is working.","type":"display","display":{"body":"The app is going through several steps:\n\n1. Analyzing the program and organization and summarizing the program's likely elements and stakeholders based on the information you provided. \n2. Customizing a logic model and evaluation framework. \n3. Creating a draft evaluation plan including project roles and some draft meeting agendas. \n\nWhen it is finished, you can copy the plan into a Word or Google document for editing. \n\nThe Evaluation Planner is based on the LogicalOutcomes Evaluation Planning Handbook at:\n\nhttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=4815131\n\nIf the evaluation plan is unhelpful, it's probably because it didn't have enough accurate information to work with. Gather a more detailed program description and try again. You can upload a program manual if you have one, for example. ","featuredImageUrl":""},"loggingEnabled":false,"outputVariable":"waiting"}},"functions":{}}}