The PART

Recommendations for the Obama Administration on Program Performance and Evaluation

Heather Rieman — Thu, 08 Jan 2009 14:54:00 -0500

In less than two weeks, President-elect Barack Obama will be sworn in to office. As the Obama Administration determines its approach to assessing program performance, we hope that they learn from the current PART process. Our recommendations include:

Create buy in, especially from Congress

If President-elect Obama truly wants to create an effective way to assess program performance, he must have buy-in from Congress, OMB, federal agencies, stakeholders and the public. Congress and other stakeholders do not trust PART as it currently stands. It is considered a subjective process that can vary significantly from reviewer to reviewer and across agencies. Future program performance or evaluation systems must ensure fair, impartial, and evenly applied outcomes across all programs and the systems should have the support of agencies and stakeholders across the political spectrum.

PART is currently run by the executive branch with little input from the legislative branch - the body that ultimately decides program funding. But it is possible to create a system that allows for Congressional input, either early on as the system is created or throughout the process. For example, the process could involve the Government Accountability Office, the Congressional investigative arm with experience in performance and accountability, in partnership with the Office of Management and Budget. Or Congress could write into legislation some or all of the indicators used to judge a particular program's performance. As such, Congress's intentions would be clear and program staff and stakeholders would be informed of the measures to be used to assess them.

Ensure staff capacity and expertise

The Obama Administration must ensure that the federal government has the capacity and expertise to assess program performance across different agencies and programs. Well constructed assessments take time and expertise to implement and can easily overextend OMB and agency staff. A more robust and comprehensive program performance system, therefore, must be adequately funded and staffed.

Ensure assessments are appropriately tailored to the program they assess

While having a common assessment for government programs can be useful, the one size fits all approach does not always capture the wide variance in missions and expected outcomes across federal programs. Measuring the performance of a Department of Defense weapons program is obviously different than assessing programs that provide human services, especially education. Future program performance systems need to take into account the type of service being provided and make sure that the corresponding assessments accurately and fairly capture the results. Other types of assessments need to be used for programs where experimental design studies are problematic or small programs where the costs of a study would be prohibitive.

Invest in a research agenda

A large share of a program's score on the PART is determined by whether or not it has demonstrated results through a high quality evaluation. Unfortunately, many programs have not participated in the research and evaluation necessary to determine whether they are effective or not - particularly Department of Education programs. As a general rule, programs on the ground do not have the resources needed to conduct larger scale evaluations. In order to properly assess a program's performance, the federal government needs to invest more money in research. Increasing funding for high quality research at the federal level will guide policymakers to more effectively use taxpayer dollars.

The future of PART

After President-elect Obama is sworn in on January 20th, he will have an opportunity to focus on program performance and eliminate programs that don't work. While we don't have details on the new Administration's plans for assessing program performance, we do have some clues. The initial ideas put forward by the Obama-Biden team include:

Create a team within the White House to focus on program performance and results for federal programs. This "SWAT team" will be lead by a new Chief Performance Officer who will report directly to the president.
"Fundamentally reconfigure" the Program Assessment Rating Tool. The performance measurement process will be opened up to "the public, Congress, and outside experts" and ideological goals will be changed.
Implement consequences for program performance. Performance teams will be sent in to reform programs that don't work. This may be accomplished by replacing management, shifting resources, or implementing improvement action plans. In some cases, programs may have to be eliminated.

Evaluating the effectiveness of federal programs is an incredibly complex task. We hope that President-elect Obama will usher in a new era in program assessment that will build on and improve the current PART process.

This is the last post in our series on the Program Assessment Rating Tool (PART). We will continue to follow developments with PART and program assessment under the new Administration. Please let us know if you have any questions or feedback!

Pros and Cons of the Program Assessment Rating Tool

Heather Rieman — Thu, 18 Dec 2008 21:52:00 -0500

Since it's inception in 2002, the Program Assessment Rating Tool (PART) has received both criticism and praise from stakeholders, researchers, and Congress. Let's explore the pros and cons of the PART:

Pros

The PART increased attention paid to program results.

Since the PART was initiated in 2002, stakeholders have paid increased attention to program performance. Department of Education budget analysts and program staff, as well as staff from OMB, have worked together to assess individual programs. As a result, many programs have created short term and long term performance measures that allow them to continually focus on demonstrating results.

The PART assessment asks some useful questions that highlight program management and design issues.

Overall, the PART assessment asks some useful and important questions. Examining 93 programs at the Department of Education and over 1,000 programs government-wide through the four PART sections (Purpose & Design, Strategic Planning, Management, and Results) has helped to reveal weaknesses and capitalize on strengths. PART assessments have also provided Congress with direction when re-authorizing legislation and help the Department of Education improve strategic planning and management.

PART made performance information more transparent and easily accessible.

OMB has made PART information transparent and accessible to the public. The OMB website has background and guidance information on the PART. OMB has also created a website, ExpectMore.Gov, which includes PART assessments for every program with a PART rating. All information is stored in a database that is searchable by program, agency, and rating. It includes annual and long term performance measures as well as program improvement plans.

Cons

The PART is a blunt instrument with a one size fits all approach.

The Program Assessment Rating Tool assesses different types of programs across all federal agencies. While it allows for a small amount of customization based on program type, it's primarily a one size fits all approach that overlooks the nuances involved in program design and implementation. The questions on PART don't always work well for Department of Education programs, particularly those that are small or designed with flexibility in mind.

The PART is dependent, in part, on research that is not available.

The PART rating is partially based on program evaluations that demonstrate effectiveness. Unfortunately, many Department of Education programs have not participated in the rigorous evaluations required for credit on a number of PART questions. Such evaluations are often expensive and time consuming, and the Department of Education has not provided the funds to support them. This is one of the reasons that more than half of the education programs are rated "Results Not Demonstrated."

The PART presents capacity challenges.

Assessing program performance takes time. PART assessments require staff from the Department of Education as well as the Office of Management and Budget. This can be a time consuming process, especially for OMB analysts who oversee assessments for many different types of programming. A GAO report found that the PART substantially increases the workload for agency and OMB staff, even after reviewers are familiar with the process.

The PART assessment is subjective, inconsistent, and not fully trusted.

Congress and other stakeholders do not necessarily trust the PART. PART reviews are conducted by different reviewers within and across different agencies, creating inconsistent results. Because it's a subjective process, it is also subject to political bias. Because the PART is an executive branch initiative that doesn't have Congressional buy in, Congress rarely uses it to inform budget decisions.

Measuring program performance is a difficult and complicated task, especially for federal government programs that perform a wide variety of services. While the PART can be improved in many ways, its creation and implementation is a step in the right direction.

Up next in the Ed Money Watch PART Series: Recommendations for the New Administration on Program Performance and Evaluation

Child Care Access Means Parents in School (CCAMPIS)

Heather Rieman — Thu, 11 Dec 2008 14:00:00 -0500

The CCAMPIS program provides funding to colleges for childcare services for low-income parents. In 2004 CCAMPIS received a "results not demonstrated" rating. In 2007 it was "re-PARTed" and received an "adequate" rating.

Let's examine how CCAMPIS fared on each section of the PART assessment:

Program Purpose and Design - CCAMPIS Score: 2004 - 80%, 2007 - 80%

CCAMPIS received full credit for four of the five questions on both the 2004 and 2007 assessments. However, both PART reviews deemed the program's design flawed, limiting its effectiveness. At the time, the statute stated that only schools that received $350,000 or more in Pell Grant money were eligible for CCAMPIS funding and limited the size of the CCAMPIS award based on that amount. Additionally, the statute dictated that the program only report data every 18-36 months as opposed to annually. (Both of these limitations were changed in the 2008 reauthorization of the Higher Education Act.)

Strategic Planning - CCAMPIS Score: 2004 - 62%, 2007 - 75%

CCAMPIS received full credit for five of the eight questions in this section on the 2004 PART review. In 2007, it received full credit for six of eight questions. Because CCAMPIS had recently set annual and long-term goals in 2004, it could not demonstrate progress towards them on the PART conducted that year. In 2007, it did receive credit for progress it subsequently achieved. Both the 2004 and 2007 PART assessments found that the CCAMPIS program did not conduct evaluations on a regular basis (though they mention that a study is underway) and that budget requests were not specifically tied to accomplishment of performance goals.

Program Management - CCAMPIS Score: 2004 - 70%, 2007 - 90%

CCAMPIS made significant progress in this section. On the 2004 assessment, the program received credit for seven of the 10 questions. The assessment found that the program didn't hold the partners accountable for results, did not have procedures to measure efficiency, and did not display performance information in a transparent and public way. In contrast, the 2007 assessment found that CCAMPIS held managers accountable for results and made performance data publicly available. However, CCAMPIS still did not have adequate procedures to measure efficiency by 2007.

Program Results/Accountability - CCAMPIS Score: 2004 - 16%, 2007 - 25%

In 2004 and 2007, CCAMPIS received points for two of five questions. But the 2007 PART found that the program met its annual performance goals by a "large extent" instead of a "small extent." After annual measures were established in 2004, CCAMPIS was able to demonstrate progress by 2007. While the program did not have any high quality evaluations in 2007, the PART mentions a forthcoming study on the need and availability of child care services at institutions of higher education.

The PART Score and Improvement Plans

Scores from the four categories are weighted to come up with a final score. (Program Purpose 20%, Strategic Planning 10%, Program Management 20%, and Program Results 50%). In 2004, the CCAMPIS was rated "Results Not Demonstrated" most likely because it lacked baseline information or performance data. In 2007 it received an "Adequate" rating. The program increased its score in three out of four sections and demonstrated progress on annual performance measures established in 2004.

CCAMPIS' improvement plan includes working with Congress to require the program to collect data annually and improving technical assistance and data reliability. As mentioned above, HEA reauthorization included many of these programmatic changes.

What did we learn from the CCAMPIS PART Assessment?

Programs can improve their PART scores over time. When CCAMPIS was re-PARTed in 2007, it improved in three out of the four sections and received a better rating overall. Most programs at the Department of Education have only gone through the PART process once.
Some programs do not have the information necessary to answer PART questions. During its initial review, CCAMPIS was not fully prepared or designed to answer the questions developed for PART. Like many other education programs, CCAMPIS did not initially have "specific" and "meaningful" performance measures, ambitious targets and baseline data, independent evaluations, or efficiency measures. As a result, CCAMPIS, and other programs, are unable to receive PART credit in many cases.
Programs are assessed on their ability to demonstrate results from "independent evaluations of sufficient scope and quality" even though many programs don't have such evidence. CCAMPIS did not receive credit for the two questions related to research demonstrating program effectiveness. CCAMPIS, like many education programs, never participated in such a study and therefore was never given an opportunity to demonstrate results, one way or another. It seems unfair to penalize a program without an evaluation when the Department of Education has not initiated or funded one.
High quality studies take time and money. CCAMPIS' 2004 review says that a study would be completed in fall of 2005. However, no such study in currently available. If programs are to demonstrate effectiveness as a component of PART, they need the time, funding, and support to develop and implement a high quality study.

Up next in the Ed Money Watch PART Series: What Works in the PART Assessment

Safe and Drug Free Schools PART Assessment

Heather Rieman — Thu, 04 Dec 2008 14:18:00 -0500

Over the past few weeks we've been looking at the PART on a macro-level. Now it's time to take a closer look at PART results for an individual program. The Safe and Drug Free Schools program (SDFS) is a block grant that provides funding to states and school districts for drug and violence prevention in schools. In 2006 it was given a "results not demonstrated" rating. Why?

Let's first examine how SDFS fared on each section of the PART assessment:

Program Purpose and Design - SDFS Score 60%

The program received full credit for three of the five questions in this section - whether the program's purpose is clear, whether the program addresses a specific problem, and whether the program is duplicative of other programs.

It received no credit for two questions. The PART assessment found that SDFS has a design flaw that limits its effectiveness. Because SDFS is a block grant, funding is thinly distributed across all 50 states. As a result, "two-thirds of all school districts receive allocations of less than $10,000, amounts typically too small to mount comprehensive and effective drug prevention and school safety programs." Though the PART points out the design problem, at the end of the day, this is really a funding issue.

The PART assessment also found that SDFS does not target resources effectively. States allocate funds based on Title I funding and enrollment, not on the prevalence of drugs or violence in specific districts or schools.

Strategic Planning - SDFS Score 62%

SDFS received full credit for five of the eight questions in this section, including whether the program has annual and long term performance measures. However, SDFS received no credit for "ambitious targets and timeframes" for either the annual or long-term measures. While SDFS does have targets for five of its seven goals and is currently conducting research to set targets for the final two, their assessment is not complete.

SDFS also lost points because its budget requests are not explicitly tied to the accomplishment of performance goals and are not derived from estimates of what is needed to achieve these outcomes. As discussed in a prior post, it is unclear whether any program's budget requests are really tied to performance goals. This criticism is not one unique to SDFS.

Program Management - SDFS Score 78%

SDFS received full credit for seven of the nine questions in this section including whether the program has strong financial management, is coordinated with related programs, engages in oversight activities, and spends funds in a timely manner. SDFS lost credit because it does not collect timely and credible performance information and does not make performance data available to the public in a transparent way. While the explanation indicates that SDFS does collect some performance information, it does not provide baseline data by which to compare current data.

Program Results/Accountability - SDFS Score 8%

SDFS received a "no" or "not applicable" on four out of five questions in this section. The program did receive a small amount of credit for meeting some of its long term performance targets. (Oddly, the program met three out of five goals for both annual and long term performance, but only received credit for the long term section.)

SDFS received no credit for the question pertaining to evidence of effectiveness from independent evaluations. While a 2001 RAND study found the program's structure was flawed, it did not assess the program's effectiveness. A new study will assess whether grantees use research-based prevention programs and is implementing them according to the research. Still, it's not clear whether this study will ultimately assess the program's effectiveness.

Most Department of Education Programs fare the worst on this section. Only two programs out of 93 received 100 percent in this section, and only 17 of the 93 programs scored better than 50 percent.

The PART Score and Improvement Plans

The scores from the four sections are weighted to produce a final score. Program Purpose makes up 20% of the total score, Strategic Planning is 10%, Program Management is 20%, and Program Results is 50%. When programs, such as SDFS, lack annual or long term performance goals, baseline information, or performance data, they are rated "Results Not Demonstrated." Almost half of the Department of Education's programs receive this rating.

Each program rated by the PART receives an improvement plan. SDFS's plan includes providing training to states on the quality and use of data, posting performance data, and working with Congress to authorize a more effective program.

What did we learn from the Safe and Drug Free Schools' PART Assessment?

PART can be useful in highlighting issues with program design or targeting. The PART can reveal weaknesses in program design that can be changed by legislation.
PART can be useful in identifying planning and management issues. The PART can help reveal weaknesses in planning and management that the Department can address.
There is limited opportunity for partial credit or work that is in progress. The PART takes a snapshot of a program at a specific time, but does not capture ongoing work or give partial credit for progress.
Education programs often score poorly on the Results section, which makes up 50 percent of the PART score. Many Department of Education programs do not have rigorous evaluations or performance information on annual and long term measures as required by this section. The 93 education programs with PART assessments score an average of 25 percent in the Results section (compared to 63 percent in Planning, 73 percent in Management, and 84 percent in Program Purpose and Design).

Up next in the Ed Money Watch PART Series: The Child Care Access Means Parents in Schools (CCAMPIS) PART assessment.

Congress and the PART

Heather Rieman — Thu, 20 Nov 2008 14:00:00 -0500

While PART ratings have a limited affect on the President's budget request, it appears they have almost no affect on the actual funding levels Congress appropriates. Programs rated "ineffective" largely continue to receive funding, while programs rated "effective" do not necessarily get increases in funding.

Why ?

Many people in Congress are unaware of the PART or have limited information about it. We called several congressional staff and asked whether they thought the PART ratings were useful and whether they were actually used to make decisions about funding levels. The majority of staffers told us that they either didn't know what the PART was or had only limited information about it. Committee staffers were more aware of the PART, but did not always have a clear sense of what it actually was or feel that it was particularly useful. If Congress doesn't have a good understanding (or in some cases even know) of PART, it is difficult for them to factor it into its funding decisions.

PART results come from the executive branch and in Congresses eyes are often considered to be subjective. The PART is run by the Office of Management and Budget, which is part of the Executive branch under the control of the President. While PART results are reported to Congress, Congress does not play a role in implementing the ratings. Additionally, as we've mentioned in the past, PART is not an entirely objective process and may be vulnerable to ideological influence. One staff person we spoke to called the PART a "political ploy" used to justify programs the Administration supports. Because Congress is removed from the PART process and may not trust its results, they are less likely to use it.

Influenced by their constituencies, their beliefs and their politics, Members of Congress will often support particular programs whether or not they are found to be effective. Many members of Congress have specific programs or policies that they champion as a result of their constituencies, their politics, or their beliefs. Even if Congress was fully aware of the PART and trusted its results, it is unlikely that members would abandon their favorite programs based on a bad PART review. Because many programs have strong constituencies of beneficiaries and service providers, it is difficult for Members to cut funding without political repercussions.

Some members of Congress have spoken about and do use PART results on occasion - when it serves their purposes. Primarily, Members use PART to provide evidence for increasing funding for programs they support, and cutting funding for programs they don't. A report from the Journal of Public Budgeting and Finance examined the congressional record to see how often Congress actually used the PART. In the 109^th Congress, members of the House Committee on Education and the Workforce and the Senate Education Labor and Pension Committee discussed PART results in only 2 percent of hearing reports.

A number of bills supporting or refuting the PART have been introduced in Congress. In 2005, Representative Todd Platts (R-PA) introduced the Program Assessment and Results Act (PARA). The bill would have required OMB to review programs every five years in a format based on the PART.

In 2006, a 2007 appropriations bill from the House Appropriations Committee on Labor, Education, and Health and Human Services included language that prohibited the use of funds to conduct PART reviews on programs administered by those agencies unless Congress specifically approved a study. Ultimately Congress rolled this appropriations bill into an Omnibus Appropriations Bill without the added language/condition.

While Congress hasn't used PART much in the appropriations process, PART reviews are sometimes considered during the reauthorization process. After undergoing a PART review, each program is given an improvement plan with recommendations on how to strengthen management and performance. These recommendations sometimes include working with Congress to strengthen the statute or change the program design.

During the recent reauthorization of the Higher Education Act, for example, information from the PART reviews of TRIO and the Federal Family Education Loan program were considered during debates surrounding the legislation.

The upcoming reauthorization of the No Child Left Behind Act provides a good opportunity to use information from PART to strengthen programs and improve accountability. Similarly, Congress may want to consider building some of their own accountability measures into the new legislation. That way Congress can ensure that evidence exists to inform PART reviews and help direct how programs are assessed by PART and other evaluations. Ultimately, Congress may be more likely to pay attention to PART results if they play a role in developing the assessment framework.

Up next in the Ed Money Watch PART Series: An in-depth look at an individual program's PART assessment

PART Results and the President’s Budget

Heather Rieman — Thu, 13 Nov 2008 14:00:00 -0500

The Program Assessment Rating Tool (PART) was created as part of the Bush Administration's Budget and Performance Integration Initiative to link program goals and performance measures to the budget process. According to the Office of Management and Budget (OMB), PART results "will be used to make decisions regarding budgets and policy."

To what degree has this actually happened? Do PART results for Department of Education programs correspond to funding levels in the President's Budget Requests?

As noted in last week's post, almost half the programs at the Department of Education do not present enough information to determine program effectiveness and receive a PART rating. But limited evidence exists to suggest that a program's PART rating, even when available, is considered when making funding decisions.

While there may be some connections between PART ratings and budget requests, they are not always consistent. For example, the President's 2009 budget did eliminate funding for three of the four Department of Education programs rated "ineffective" - Even Start, the Perkins Loan Program, and Vocational Education State Grants. But the budget did include level funding for TRIO Upward Bound, which was also found to be "ineffective" (though some prior year requests did try to cut funding).

The President also proposed eliminating 44 additional programs at the Department of Education. Of the 18 programs with PART reviews, two were rated "adequate," and 16 were rated "results not demonstrated." The other 26 programs targeted for elimination had not gone through a PART review to determine program effectiveness.

The President's 2009 budget request did include funding increases for four of the five programs in the Department of Education rated "effective" - Institute of Education Sciences Research, National Assessment for Educational Progress, National Center for Education Statistics, and Reading First State Grants. The President also requested increases for two programs rated "moderately effective," nine programs rated "adequate," and two programs rated "results not demonstrated."

The connection between PART ratings and the President's funding request is inconsistent. While PART may be used to justify a funding increase or decrease in some cases, it is not always the case.

This is partly by design. OMB points out that funding requests are not solely based on the PART. A program's PART rating is only one factor among many used to inform budget recommendations. A bad rating does not necessarily mean decreased funding and a good rating does not guarantee an increase.

Looking at the list of programs rated "effective," we wonder if ideology played any role in the ratings. Three of the six programs currently rated "effective" focus on research, something the Administration has strongly supported since taking office. Over the years, the Administration has also requested funding increases for Reading First, one of their signature programs.

Despite attempts to ensure that the PART is as unbiased as possible, the PART is ultimately a subjective tool. As a result, it may reflect the biases of those conducting the assessment - the Executive Branch.

Making funding decisions based on solid assessments of program effectiveness is a worthwhile goal. PART is a step forward in linking program performance to the budget process. Still, questions remain about the reliability of the PART instrument itself and the extent to which it is actually used to make funding decisions.

Up next in the Ed Money Watch PART Series: PART Results and Congress

PART Assessments of Department of Education Programs – The Results

Heather Rieman — Thu, 06 Nov 2008 14:00:00 -0500

During the presidential campaign, both Senator McCain and Senator Obama talked about eliminating ineffective federal programs. We commend their emphasis on funding programs that show results. But what evidence will President-elect Obama use to actually determine which programs are working and which aren't?

One source of information the president could turn to for guidance is the Program Assessment Rating Tool (PART). But for many programs, particularly at the Department of Education, President-elect Obama might find only limited information to determine whether some programs are actually effective.

Results for Department of Education Programs

Since 2002, 93 Department of Education programs have gone through the PART review process. Of those, 44 were rated "results not demonstrated." That means nearly half the programs did not have acceptable performance measures or lacked the performance data necessary to make a determination.

Of Department of Education programs that did have enough information for a rating, only six programs were found to be "effective" - Adult Education State Grants, Institute of Education Sciences Research, National Assessment for Educational Progress, National Center for Education Statistics, Reading First State Grants, and Transition to Teaching. Only four programs are currently rated "ineffective" - Even Start, Perkins Loan Programs, TRIO Upward Bound, and Vocational Education State Grants. The remaining 39 programs are rated "moderately effective" or "adequate."

You can visit www.ExpectMore.Gov to see individual program results. The website provides information about ratings for programs at every agency and links to their individual PART assessments and improvement plans.

How Department of Education Results Compare with Other Agencies

Examining PART results for the Department of Education compared to program results in other departments provides important perspective on the efficacy of PART for education.

Generally, departments that administer social services, such as housing, labor, and education, have a lower percentage of programs rated "effective" and more programs with "results not demonstrated" than other agencies. Of all the cabinet level agencies, the Department of Education has the highest proportion of programs without demonstrated results. (The second highest is Veteran's Affairs, which has 30 percent of the programs rated "results not demonstrated.") There are a variety of reasons for this disparity.

Social service programs, particularly in education, can be hard to evaluate. Often, data is not available to provide evidence for a PART rating because many programs at the Department of Education have not undergone rigorous studies or evaluations. For those with available data, a scientifically controlled experiment may not be possible due to the nature of the services provided or because such a study would be time consuming and expensive. Additionally, many programs require longitudinal data and multiple cohorts to conduct rigorous research and such data systems, where they exist, are still too young to be relevant.

We find it problematic that so many education programs undergo PART assessments without evidence to either support or refute them. If PART is meant to be a litmus test for continued program funding, this dearth of research must be remedied. As we have talked about in past posts, this has significant implications for the Department of Education research budget. In fiscal year 2008, the Department of Education dedicated only one half of one percent of its budget to research and development.

The size and type of education programs examined under PART may also affect program results. The Department of Education has many small programs with limited staff and funding. Capacity often limits a program's ability to collect or monitor annual and long term performance data.

Larger programs at the Department tend to be competitive or block grant programs. A report from OMB Watch suggests that competitive and block grant programs do not perform as well as other types of programs on the PART. Block grants distribute funding to local districts, making it difficult to monitor and evaluate results centrally. More than a third of Department of Education programs with PART ratings are block grant programs, and 84 of the 93 programs were either block grant or competitive grant programs - a higher percentage than in many other agencies.

PART is supposed to be an unbiased and accurate assessment of federal programs. But answering the PART questions requires some degree of subjectivity. As a result, the PART review process can be vulnerable to ideology or unreliability across reviewers or agencies. Similarly, PART's one-size-fits-all approach to assessing federal programs may not accurately capture the nuances in different types of programs, disadvantaging Department of Education and other social service programs.

President-elect Obama talked about using a scalpel, not a hatchet, to eliminate specific programs that don't work. But what will he use to guide his scalpel? How will he know which programs are working and which aren't? In the end, President-elect Obama is likely to find that PART provides only limited guidance when determining which programs are and are not effective, especially when it comes to education.

An Overview of the Program Assessment Rating Tool (PART) and How it Works

Heather Rieman — Thu, 30 Oct 2008 15:00:00 -0400

Last week we introduced our blog series on the Program Assessment Rating Tool (PART), a questionnaire designed to assess and rank the effectiveness of federal programs. Since 2002, the Office of Management and Budget has used PART to examine hundreds of programs in various agencies in the federal government. To date, more than 1000 programs have been through the PART process, including 93 programs at the Department of Education.

How it Works

The PART questionnaire is comprised of approximately 25 standard questions, as well as a few additional questions tailored to the type of program being evaluated. Programs can belong to one of seven categories: direct federal; competitive grants; block or formula grants; regulatory; capital assets and service acquisition; credit; and research and development.

The questions are divided into four categories:

Program Purpose & Design: Asks questions about the clarity of the program's purpose, the problem the program addresses, whether it is well-designed, and whether it is duplicative of other programs.
Strategic Planning: Asks questions about whether there are annual and long-term performance goals with ambitious targets and timeframes, whether the budget is aligned with program goals, and whether there are independent, high-quality evaluations of the program conducted on a regular basis.
Program Management: Asks questions about whether programs use performance information to manage the program, if people are held accountable for performance results, whether there are measures for cost effectiveness and strong financial management practices, and whether management deficiencies have been addressed.
Program Results: Asks questions about whether programs achieve their annual and long-term performance goals, how the program compares to other similar programs, whether the program is cost effective, and if independent evaluations of the program indicate the program is achieving results. (See below for a list of all 25 questions.)

How Programs Are Scored

Most assessment questions are simple yes or no questions, however the Program Results section does allow for additional responses (yes, large extent, small extent, and no). "Not applicable" is also an acceptable response for some questions. The scores from the four categories are weighted to come up with a final score. Program Purpose & Design makes up 20% of a program's total score, Strategic Planning is 10%, Program Management is 20%, and Program Results is 50%.

The Office of Management and Budget, in coordination with federal agencies, scores each program on a scale of 0-100. This numeric score is then translated into a qualitative rating of "effective" (85-100), "moderately effective" (70-84), "adequate" (50-69), or "ineffective" (0-49). Programs that do not have long term objectives or performance information are rated "results not demonstrated."

The PART is often mistaken for a full research evaluation - a study conducted by researchers that measures program effectiveness. The PART is actually not a research study, but a rubric used to assess how a program is designed and implemented, and whether any evidence of effectiveness exists. However, full research studies can be used to answer PART questions regarding program results. Programs rated "results not demonstrated" are not necessarily ineffective, it just means that evidence does not yet exist to support or refute its effectiveness.

PART results are released with the President's Budget request each February and are intended to inform future program funding. Results are also used to shape program improvement plans. Programs can request to repeat the PART process and improve their ratings if they have demonstrated recent progress or have new information available.

Up next in the Ed Money Watch PART Series: PART results for Department of Education programs.

____________________________

Basic questions in PART:

Section I. Program Purpose and Design

1.1: Is the program purpose clear?

1.2: Does the program address a specific and existing problem, interest, or need?

1.3: Is the program designed so that it is not redundant or duplicative of any other Federal, State, local or private effort?

1.4: Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

1.5: Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Section II. Strategic Planning

2.1: Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

2.2: Does the program have ambitious targets and timeframes for its long-term measures?

2.3: Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

2.4: Does the program have baselines and ambitious targets for its annual measures?

2.5: Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

2.6: Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

2.7: Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

2.8: Has the program taken meaningful steps to correct its strategic planning deficiencies?

Section III. Program Management

3.1: Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

3.2: Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

3.3: Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose, and accurately reported?

3.4: Does the program have procedures to measure and achieve efficiencies and cost effectiveness in program execution?

3.5: Does the program collaborate and coordinate effectively with related programs?

3.6: Does the program use strong financial management practices?

3.7: Has the program taken meaningful steps to address its management deficiencies?

Section IV. Program Results/Accountability

4.1: Has the program demonstrated adequate progress in achieving its long-term performance goals? 4.2: Does the program (including program partners) achieve its annual performance goals?

4.3: Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

4.4: Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

4.5: Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Evaluating the Effectiveness of Federal Programs: An In-depth Look at the Program Assessment Rating Tool and Education Programs

Heather Rieman — Tue, 21 Oct 2008 21:21:00 -0400

We all want to know if the government is spending our tax dollars wisely: Are funds being used on education programs that are actually achieving results? Or are they being spent on ineffective programs? Which programs are working? Which aren't? Results matter.

Most people would agree that these are important questions. But actually evaluating the effectiveness of federal programs is an incredibly complex task.

In 2002, the Office of Management and Budget launched an initiative to help measure program effectiveness and provide guidance to the policymaking, budgeting and appropriations process. The resulting Program Assessment Rating Tool (PART) is designed to "assess and improve program performance so that the Federal government can achieve better results."

Since the inception of PART, 1017 programs across 15 Departments have been reviewed, 93 at the Department of Education alone. Programs are assessed and assigned points in four categories: Program Purpose & Design, Strategic Planning, Program Management, and Program Results. At the end of the process, programs are rated "effective," "moderately effective," "adequate," "ineffective," or "results not demonstrated." The President's budget request to Congress often reflects the results of different PART assessments.

The Federal Education Budget Project aims to heighten the quality of the debate on education funding. Because PART stands as the only formal and systematic assessment rubric, we think an evaluation of the process itself is a worthwhile endeavor. Over the next few months, Ed Money Watch will feature a series of posts exploring, explaining, and at times critiquing, the Program Assessment Rating Tool.

We hope to educate readers about the PART process, consider the extent to which evaluations affect funding, program improvement and elimination, assess the strengths and weaknesses of the PART process, and provide suggestions and recommendations for PART improvement. We will also examine some of the more controversial evaluations to date.

Dedicating scarce federal funding to effective education programs is of vital importance to the economy and America's education agenda. We hope that our series will ignite a conversation about lessons learned from the current PART process and how the incoming administration can improve not only PART, but education programming as a whole.

We look forward to hearing from you during the series and hope that you will take the time to share your thoughts, suggestions, and critiques in our reader comments section.