National Quality Improvement Center on the Privatization of Child Welfare Services Request for Applications October-November 2006
Questions and Answers
October 30, 2006
Q1: If awarded, what is the time frame of the research period?
The implementation period for delivery of services will be for three years, January 1, 2007 through December 31, 2009. After this, projects will continue to receive funding through federal fiscal year 07/08 to complete their data analysis and final report, and to engage in dissemination efforts.
Q2. How far back will data collection go?
This will be determined by each site’s own implementation schedule, size of population served and evaluation plan. For instance, if a site is able to implement an experimental, random assignment evaluation design, collection of historic data may not be required. If a site is using a same time, cross site comparison (e.g. comparing counties) the use of baseline (pre-implementation data) for the treatment group would strengthen the design, but is not required. However, for sites that are using a quasi-experimental, pre/post comparison, sites would be encouraged to collect data for the same length of time pre-implementation as post implementation or at a minimum, collect enough historic data to have a sufficiently large sample to be able to discern statistically significant differences between the two groups.
Q3: If awarded, what would be the agency’s requirements/mandates/expectations?
The public agency is required to be the lead agency. The lead agency as well as evaluation personnel would be required to attend project meetings twice a year, monthly conference calls as well as host two site visits by QIC PCW staff twice in Year1 and annual site visits in the following years. These site visits would involve all project staff (private provider partners, university partners, agency staff, etc.). In addition, the agency will be required to fulfill the specific requirements outlined in the RFA regarding implementation of the project and a detailed work plan to be negotiated with the public agency, complete a site-specific evaluation, compliance with the cross-site evaluation activities to be negotiated with funded projects, and share data and products with the QIC.
Q4: What "reports" are expected?
Each year, the lead agency is required to submit two reports, which in turn, will be a part of the QIC PCW semi-annual reporting to the Children’s Bureau, in addition to a final report following the three year implementation period. In addition, part of the cross-site evaluation design to be negotiated with projects will involve the submittal of data/findings as is appropriate.
Q5: What will be the format of these reports?
The Children’s Bureau is currently working on the development of this reporting format and will be provided to the projects as they begin their work. A draft format will be provided to projects for comment and revision prior to the first reporting period.
Q6: Will there be ongoing phone conferences?
The QIC PCW Staff will coordinate monthly conference calls with all projects and will also conduct conference calls as needed such as with project evaluators when group discussion and problem-solving is warranted.
Q7: Are there any specific mandated initiatives?
The RFA is not mandating a particular model of a performance based contracting and quality assurance system. However the RFA does detail a number of applicant requirements on p. 5-6.
Q8: If a state has not released the details of a new roll-out plan or contract publicly or to their providers at the time of submission of the RFA, should the state leave out those confidential details in their application?
Confidential information can be included in proposals. The application submitted is not the property of the QIC, and the QIC will not be making information from applications submitted public during the review process. Once awards are made, information on successful applications will be summarized for release, but entire applications will not be made public. If an open records request were made, only successful applications would be eligible for release, and they will have the opportunity for redaction.
It is critically important that the reviewers have the benefit of the detail of the existing system and the proposed model for it to be accurately and fairly reviewed. External Reviewers will be trained and be signing a statement of confidentiality prior to the review and will therefore not be divulging confidential information from applications. It is recommended that the applicant consider the application confidential, and provide detailed information in the application. The review process will not include the opportunity for applicants to submit addendums or follow up information prior to the award decision-making.
Q9: Kansas is in our 3rd set of contracts, starting July 1, 2005, and therefore wouldn't be proposing a new project. What is the expectation concerning this?
The expectation is that your agency would be proposing an innovative practice within the topical focus area: performance based contracting and quality assurance systems. It is important for you to focus on the QIC needs assessment and knowledge gaps analysis findings in order to create an innovative approach and/or enhance a current system and/or build on what is already in place. There are really two aspects of what we are hoping to test: the performance based contract and its components (which for some states may already be in place) and the quality assurance process that is designed to move the state toward achievement of outcomes for families and children.
Q10: What impact would it have if historical information regarding a comprehensive, inclusive, planning process is limited (Research Question #1)? Given the number of years that have passed since the inception of privatization in this state, such information would not be readily available.
Applicants are expected to address all five of the research questions to the best of their ability. In situations where project planning took place in the past, successful applicants will be expected to describe retrospectively the extent and nature of planning activities and, to the best of their ability, the impact that this planning had on the project model and implementation efforts. If additional planning is undertaken to enhance or augment this effort, then information and data on the inclusivity and comprehensiveness of this process should be documented and its impact on outcomes should be analyzed.
Q11: What are the expectations concerning the information on page 6 that indicates the Center is looking for grantees that are at similar stages of implementation?
Applicants are required to have already privatized some component of their out-of-home care program within the geographic area proposed for the project and primary (or shared) case management rests with the private provider. They need not have already instituted a performance based contracting system but may have. Given these eligibility requirements, the length of implementation will be taken into account in terms of how it relates to comparability of projects and contribution to the knowledge base when award decisions are made.
Q12: Quasi-experimental design: Kansas can compare changes over time but since the whole state is privatized and the contracts are the same, it would not be feasible to compare one county to another or one region to another. What is the expectation about this?
Applicants are expected to implement the most rigorous approach that is feasible to evaluate the impact of their PBC/QA model. While a pre/post comparison is the weakest quasi-experimental approach methodologically, we understand that this might be the best that can be done in certain situations where the program model has been implemented statewide. If new components are rolled out on a pilot basis, they could benefit from a same time comparison.
Q13: What is the possibility of focusing the project to just one area of practice; specifically, balancing appropriate levels of systemic and case level review without micromanagement in contract monitoring?
As long as the site has a performance based contract in place that is being monitored for success, applicants can choose to focus their innovation on a particular aspect of this system which they believe is innovative and deserving of evaluation. The RFA includes a number of promising practices on p. 4-5 and applicants are required to incorporate at least one into their proposed PBC/QA system. Evaluation activities will focus on both PBC and QA reform activities.
Q14: Our agency develops/monitors/oversees all contracts for a particular District, does the state agency still need to be the lead agency?
Yes; this is a Children’s Bureau’s requirement as the public agency is the one who is held accountable for the provision of mandated child welfare services and the outcomes, regardless of who is actually contracted to provide the services. However, it is acceptable under this project for the public agency to subcontract certain roles and responsibilities to a private provider, university, etc. These roles and responsibilities must be clearly defined. In such systems where “umbrella” agencies provide the subcontracting role to individual private providers, it would be appropriate to include the public agency as well as this umbrella agency and individual providers in the application.
Q15: Is the lead agency (public agency) then considered the project director?
Yes. However, this does not preclude the public agency from subcontracting substantial responsibility for the implementation of the project and the evaluation to a university or private provider. The lead (or public) agency will be the subgrantee under this project and will need to approve project budgets, formally submit reports and provide overall direction to the project.
Q16: Can you use a portion of public agency salaries for match?
Yes, as long as that employee’s position is not Federally-funded 100%.
Q17: If three projects were not funded, could the additional funds be dispersed between the funded projects?
Yes; the Children’s Bureau requires that a specific amount of QIC funding be used for funding the research and demonstration projects; therefore, that total amount would have to be allocated to the funded projects.
Q18: Does a state/county have to currently have ongoing privatized services in order to apply?
Yes; it is a requirement that the public agency is currently, or will be by the time the project is implemented, privatizing case management responsibilities or at a minimum shared decision making for some segment of the out-of-home care population in the geographical area of the state that is the proposed site of the pilot. It is not required that largescale, system-wide privatization is in place.
Q19: What kind of qualifications do you expect the external evaluator to have?
It would be important that this person have demonstrated expertise in program and policy evaluation similar in size, scope and complexity to this effort and have conducted both process and outcome evaluation methodologies.
Q20: What criteria does the comparison group need to meet? Can you use historical data?
We are striving for the strongest, most rigorous evaluation design feasible. While an experimental design involving random assignment is strongest, this may not be reasonable for a number of reasons. A quasi-experimental design is the next strongest approach and is a requirement for the application. Within quasi-experimental design there are also varying levels of rigor. A same-time comparison within the same site is the strongest, followed by a same-time comparison of a comparable geographical area. Comparisons of separate cohorts that are not same-time are the least strong, but may be all that is feasible if the model has been rolled out state-wide. Within this hierarchy or rigor, the most relevant evaluation design should be applied. If relying on historical comparisons, it is important that there be controls to ensure that the two cohorts are separate and distinct—with the treatment group experiencing the intervention, while the comparison group does not. In other words, for analysis purposes, children in the historical comparison group should not have experienced the intervention under study.
Q21: If we do not know who will be the external evaluator for this project, can we provide backgrounds on more than one possible evaluator?
It is best if you offer the qualifications of a specific person. If this is not possible at the time of the application, and a particular firm or organization whose credentials and qualifications for completing evaluations of this sort can be identified a justification of this may be offered.
Q22: Do we need to have selected our private providers at the time the application is submitted?
Yes. You must provide a letter of commitment from all partners in your proposed project. These letters also should describe the partner’s specific role in the project. If you will be selecting from a pool, each would have to submit a letter of commitment. However, it is preferable to have partnership selection clearly delineated in the application.
Q23: Can you have a multi-layer subcontracting process (such as the public agency subcontracts with a university or “lead” private provider which in turn subcontracts with individual providers)?
Yes. It is possible but you will need letters of commitment from each agency that is being contracted.
Q24: We may have some time and procedural (i.e. board approval) limitations regarding the hiring of an external evaluator, how do you see this affecting the requirement of implementation within 90 day?
The 90-day implementation requirement means active initiation of the project regarding setting up of project staff, research design, etc. It would seem appropriate to have board or other appropriate entity approval at least in concept at the time the application is submitted. The research guidelines and plan indicate negotiation of the cross-site evaluation and refinement of the project evaluation will occur within the first few months with data collection to begin within six months, so it will be important to have key personnel on board and involved as soon as possible, even if the formal subcontracting process is requiring additional time.
Q25: Will each project have the opportunity to see the collective data from all projects?
Yes. The QIC PCW Staff will pull together data from individual projects for reports to the Children’s Bureau and will share summaries of this with all projects. They will be promoting sharing of data and implementation details by projects through project meetings and at specified times throughout the funding period as appropriate to promote information sharing and collaborative discussion and planning.
Q26: Who attends semi-annual and/or annual meetings, conferences, etc.?
We would expect that two project staff would attend the first project meeting. This would probably be someone from the lead agency and the external evaluator. Then, the lead agency could then negotiate who would attend subsequent annual project meetings, but this will involve at least two representatives from each project.
Q27: Do you have a recommendation regarding the development of the travel budget?
This amount usually comes from estimating the cost of airfare, lodging, and other travel-related expenses for two people for two meetings per year typically in the Washington, DC area.
Q28: Will data sets be the same for everyone?
No. Each site will develop its own evaluation plan including the selection of data elements with input and direction from QIC PCW staff. Also, while it is expected that each site will measure both short and long term outcomes, ultimately, all sites will want to assess the impact of their models on relevant indicators of child safety, permanency, and well-being.
Q29: Our state is transitioning to a statewide automated system in February and may impact our ability to obtain necessary data for a short while. Do you see this as a barrier?If this were to be a long-term situation it would certainly be an issue. However assuming this is a short term transition it is not a barrier that can’t be overcome. All states will probably experience challenges associated with data and data collection. It would be important, however, to engage state agency personnel to get their commitment regarding the needs of this project. The applicant should describe what data would be influenced by the transition, what back up data will be secured, and what other forms of data will remain stable throughout the period that will be relevant to the project. All projects should thoroughly and accurately describe the nature and quality of their administrative data in their application, as well as what procedures will be necessary to secure approval to access and share case level data within the evaluation process.