The basic philosophy of the reorganization of central government ministries is to convert postwar-type administration systems, which have been exhausted in many aspects during the 50 years after the war, to 21st century administration systems that would be more suitable for a free and fair society.

The policy evaluation system was newly introduced as a pillar to establish new administration systems. The new evaluation system was given three major points: accountability to the people, a high-quality efficient administration, and a results-oriented administration. The new system intends to create a scheme for reviewing policies and then reflect the result of the review on policy planning, which was not implemented effectively under the conventional administration programs.

About 10,000 policy evaluations have been conducted for the entire government of Japan, every year during the past four years. Furthermore, efforts have been made to reflect evaluation results on the budget request, which is to be submitted by the end of August of the previous year. Another successful outcome is that the percentage of the number of policies with numerical target level for performance evaluation has increased.

The committee will thoroughly implement the policy evaluation in line with key policies of the cabinet, and will strengthen the linkage of evaluations with budgeting and settling. In the meantime, as a future critical task, it is necessary to cope with the unavoidable problem that is the self-evaluation system, which lacks complete objectivity, and stress the importance of policy evaluation in such areas that concern people.

The committee should continuously watch the government's actions, and should submit messages when it is necessary.

Keynote Speech

Policy Evaluation in Japan - Problems and Solutions in six years period"
Michio Muramatsu, Professor at Gakushuin University
(Presentation Papers)

As the primary aspect of policy evaluation in Japan, self-evaluation is obligatory and its objectivity and strictness is ensured by the Ministry of Internal Affairs and Communications, and the evaluation method is flexibly left to each individual ministry. As the secondary aspect, all government personnel get involved in the evaluation, accepting evaluation as a valid part of government business. This is a very important part of this evaluation system. As the tertiary aspect, tension exists between sections in a ministry that mainly carry out policy activities and specialists in the same ministry who are engaged in policy evaluation. As the final aspect, the evaluation has been studied as a method, which is not bound by budgeting; hence the evaluation is not directly related to budgeting. This aspect is valuable in view of the nature of the evaluation.

There are three merits of Japanese policy evaluation. First, the legal scheme requires that all personnel be involved in the evaluation as a norm of government employees. Second, the introduction of policy evaluation has gives them the opportunity to strategically review the order of priority. Third, the evaluation considerably discloses to outside people what policies at individual ministry are valued and to what extent they are valued by these ministries. This has enhanced the government transparency.

At the same time, there are at least two issues to be addressed. First, it is not well seen how individual ministries utilize policy evaluation information and if they utilizes the information to efficiently manage to implement their duties. Second, there are extremely voluminous papers needed for policy evaluation. This implicates that it's not clarified what is to be done in the future in these papers.

It is said that the policy evaluation may have abolished some of projects. However, it does not seem that such policies, which were considered to be quite major but are somewhat questionable, were evaluated thoroughly. Now that policy evaluation is entering its second stage, evaluation should be taken one step further. Hopefully evaluations will be conducted thoroughly, and not as a hasty attempt to abolish some policies.

I believe that efforts should be made to expand attentive layers, the think tanks, the academia, into active layers

Special Speech 1

"Performance Regimes and Institutional Context: Comparing Japan, the United Kingdom and the United States"
Colin Talbot, Professor at the University of Manchester
(Presentation Papers(1)、(2))

Today, I would like to compare Japanese reforms to similar reforms, which have taken place in the United Kingdom and the United States, and talk about the performance regimes and institutional context. The movement we have seen in the last 25 years toward more result reporting is not new. Today, most OECD countries are carrying out some sort of result reporting. As a recent trend, it is becoming larger, broader, deeper, and longer than any previous reform efforts in this field.

I would like to focus on three systems: Government Policy Evaluation Act in Japan, the U.K. Public Service Agreement (PSA) and the U.S. Government Performance and Results Act (GPRA). In Japan, "evaluation" as a term and as an approach is often used instead of "performance." On the other hand, in the United States and the United Kingdom, there are two large academic communities divided into those who promote policy evaluation and those who promote performance management. There is also a traditional difference between the two communities, which are divided vertically. Policy evaluation focuses on programs and policies, and evaluates individual policies, both qualitatively and quantitatively. In the meantime, performance evaluation focuses on organizations and continually evaluates plural programs with quantitative methods. However, in recent performance reporting, convergence of evaluation and performance has been observed in connection with the trend that long-term outcomes are have become increasingly valuable.

Looking at institutional factors, the process in the United Kingdom is quite simple; the Treasury negotiates public service agreements regarding performance targets with line ministries who in turn negotiate performance targets with their agencies. However, as a matter of fact, various factors exist to form a ministries' performance. In particular, Parliament gets involved by carrying out scrutiny and offering advice to government ministries about performance targets. The National Audit Office (NAO), courts, health bodies and education bodies, and users as the sixth factor may perform roles in forming performance. Finally, the last category is partner organizations.

I would like to look at the differences in the roles of core ministries and legislature among the United Kingdom, Japan, and the United States and compare these countries in terms of how both of them relate to line agencies. First, there are different systems in the United Kingdom, Japan and the United States in terms of the relationship between the executive and the legislative branches. In the United States, powers are separated, and there is a power contest between Congress and the executive branch. The United Kingdom has an executive-dominated system, and Japan leans slightly further toward the separation of powers. While GPRA comes from the legislature in the United States, it comes from the executive in Japan and the United Kingdom. This difference is not surprising if the roles of the executive and the legislative branches in these three countries are considered.

Let's examine how result reporting is integrated into the budget. The United Kingdom claims that performance information is integrated into the budgetary process. However, there is no evidence that performance results are actually used in the budgetary process. Although the United States is listed at a high level of performance integration into the budgetary process, this is because the United States claims so, yet there is no clear evidence.

There are variations in how much reporting is done. While the Treasury negotiates PSA performance targets with line ministries in the United Kingdom, line ministries set up their performance targets in Japan and the United States. In the latter two countries, line ministries have great ownership of these processes.

Ministries are disclosing to the outside such information showing how they utilize performance results effectively in order to improve their performance. However, these ministries are utilizing little of the information within their organization.

Special Speech 2

Policy Evaluation: Tools to Improve Government and/or Shrink It?
Steven Kelma, Professor at Harvard University
(Presentation Papers)

The goal of policy evaluation is to change the policymaking process, i.e. to try to add a new element to the traditional policymaking process that has been dominated by political parties, interest groups, public opinion, and the like. The new element to be added is facts and evidence.

Policy development refers to activities that are undertaken when the government studies new laws, new projects, or new regulations. The activities are composed of three: (1) studies, statistics, policy arguments and research, which are input into the policy development process; (2) cost-benefit analysis for all public work projects before they can be started; and (3) review of regulations.

In the United States, "policy evaluation" is used in a more limited way than in Japan. It refers to studies to determine whether a program is working and how well the program is working, and they are often carried out as very large research projects. One of the most interesting features is that evaluations within the ministries and the role of evaluation organizations generally are more important, the more controversial the program is. As the second feature, policy evaluation has, as well as policy development, the problem of objectivity as a task. The problem of objectivity has been coped with through several steps in the United States. The first step was setting up evaluation offices as independent offices within the ministry. Second, the role of Government Accountability Office (GAO) can be pointed out as an outside audit and evaluation organization. As the third step, outsiders, such as professors and people of think tanks, are effectively utilized. It is another feature that policy evaluations are generally most influential where political leaders wish to change existing policies.

With regard to the influence of evaluation on policymaking, it can be pointed out as one of the features that conflicting evaluations and partisan evaluations are increasing. As a second feature, although it is very difficult that any particular evaluation has an impact on a particular policy, policies are affected by evaluations finally during a long period of time. As a third feature, there are two independent streams (policy stream and political stream) and, as a conclusion, "evaluation" in the United States tends to be used as a tool to shrink government.

Performance measurement is treated as part of policy evaluation in Japan while in the United States, performance measurement and policy evaluation are different. There are two goals of performance measurement: accountability and performance improvement. Performance measurement can be used by the organizational management to improve the organization in four ways. One is to provide employees with challenges to be tacked in a focused manner. A second use is to motivate people, and a third use to help people learn. A fourth use is to distribute resources within the organization.

The Bush administration has introduced the Program Assessment Rating Tool (PART) as an effort to link performance measurement and evaluation and budgets, successfully more than expected. As a feature of the PART, it mixes performance measurement and budgeting, and therefore can be used as a budgeting tool. As a second feature, the tool provides a rhetoric that a program that does not work can be cut. However, a program is failing maybe because it does not have enough money. In my opinion, people may draw conclusions too quickly about the impact of a good rating or a bad rating on what should happen to the budget of the program. Third feature is that the Office of Management and Budget likes outcome measures. As a fourth feature, the Office of Management and Budget, requesting the ministries to submit the results of evaluation as part of the PART, strongly prefers external evaluation and approves internal evaluation only if this is done by an independent bureau in the ministry. Finally, there is evidence that programs acquiring good PART score do produce increased budgets.

As a first lesson for Japan, it seems to me that the development policy evaluation in Japan is part of a cultural change in Japanese policymaking. Cost-benefit analyses used in most policy evaluation look at the total benefits and costs to society as a whole and do not look at distributional considerations. In some very controversial areas, such as public projects, agriculture, and trade policy, not only issues of economic efficiency but also issues of distribution should be considered. As a second lesson for Japan, in order that policy evaluations will be sufficiently objective, it is necessary that they are made fairly independent. For this purpose, evaluations should be made by independent units within ministries, through approval of outside analyses in such a manner that Ministry of Internal Affairs and Communications is trying to do, or by an infrastructure outside of the ministries and outside of government, such as universities and think tanks. The last point is that a priority for policy evaluation in Japan should be using performance measurement as a management tool to improve government.

As to the roles of Ministry of Internal Affairs and Communications, first, the ministry should support development of tools for performance measurement and collect best practices of performance improvement. Second, the ministry should support development of research methods and techniques in the private sector, particularly at universities for doing evaluation studies. Lastly, Ministry of Internal Affairs and Communications should collaborate with Ministry of Finance by giving performance information and analyzing performance information to assist Ministry of Finance in using the information in the budget process.

Welcome Address,

Heizo Takenaka
Minister of Internal Affairs and Communications

Policy evaluation, which is extremely important, just gets understandings among the public. However, it should be more well-known to the people. In democratic societies it is the singl e greatest way for enhancing policymaking performance to let the people know thoroughly the meanings of policies and policy evaluations that are going on.

Since 2001 in Japan, the policymaking process has changed drastically. First of all, the process of determining the basic direction of policy has become established as a "Big-boned policies", which are formulated after discussing policy intensively and thoroughly. The procedure has explicitly changed to such a process where a basic policy line is decided in June and thereafter budgeting is studied until December. In this innovative change, the foundation for evaluating and discussing policies orderly is being set up in Japan. Although policy evaluation is taking its steps forward these days in Japan, it has not yet been rooted sufficiently.

All the people attending this symposium today are highly knowledge and deeply interested in policy evaluation. I hope you will make discussions in a focused manner and communicate extensively to the public. There are many challenges that the Ministry of Internal Affairs and Communications should tackle. I hope you will give us suggestions, including strenuous ones.

Panel Discussion

"Importance and Challenges of Policy Evaluation in the 21st Century: For Development of Policy Evaluation in Japan"

○Panelists:    Uichiro Niwa
                    Yoshitsugu Kanemoto, Chairman of Subcommittee on Policy Evaluation of
                                                   Commission of Policy Evaluation and Evaluation of
                                                   Incorporated Administrative Agencies, Professor, University
                                                   of Tokyo, Graduate School of Economics, Graduated School
                                                   of Public Policy
Colin Talbo
Steven Kelman

○Moderator: Masahiro Horie, Vice-Minister for Policy Coordination, Ministry of Internal Affairs and
                                        Communications

<Challenges of policy evaluation in Japan>
(Kanemoto) One feature of policy evaluation in Japan is that all the three types of evaluation; project evaluation, performance evaluation, and comprehensive evaluation, have been introduced into all the ministries, simultaneously under the Government Policy Evaluation Act. However, the comprehensive evaluation, which corresponds to program evaluation, has not been developed significantly.

There are such problems that, first, persons who evaluate policies lack high morale and, second, policy evaluation is low in the professional analysis level.

(Talbot) For improving policy evaluation in Japan, the volume of reporting should be reviewed, and independent validation should be established.

(Kelman) There are great opportunities for improvement in the area of performance management. Since Japanese society is more trusting people, not conflict-oriented, than those of the United Kingdom and the United States, performance evaluation should be used by actual working people, not third party people.

(Niwa) I think the introduction of the policy evaluation system was extremely meaningful. Third-party evaluation would be difficult without a lot of professional personnel. External evaluation is meaningless if it is the same as self-evaluation in contents. If a secondary evaluation is made by involving a third party, then a new perspective could be brought in, such as quantification of target level of their policies. As for making comprehensible evaluation reports, it should be selective and concentrative what are included in.

(Kanemoto) I think self-evaluation is most effective for performance measurement. However, performance measurement is not valid in many areas. It would be questionable if measurement would be done only in areas that are selected by ministries based on their subjective interest. Policy evaluation aiming at a smaller government has not been done well. We should make more efforts in this area.

<Management>
(Niwa) To make government administration more efficient, it may be necessary for the political side to set up clear targets for each individual ministry and make ministers, bureau directors, and other officers have responsibility, power, and authority.

(Kelman) Performance management may not be successful for some organizations. It is important to utilize evaluation results mixed with high-level judgment because the fund must be saved even for good evaluation results or budget may be increased even for bad evaluation results.

(Kanemoto) Difficulty in improvement is in relation to the self-evaluation system adopted in Japan. Another problem is that the Ministry of Finance is in no position to get into the process of evaluation. Nevertheless, I think improvements are being made.

(Talbot) When evaluation is done, it is important to be forward-looking to evaluate organizations and programs, focusing on what to be improved. The contents of evaluation should be in the public domain. The amount of the budget and Allocating and levels of performance should not be mechanically linked. Transparent discussion should be needed on these issues.

<Publicity of the policy evaluation system, and securing and developing human resources of professional for evaluations> (Niwa) People might be able to understand the importance of policy evaluation if topics that are familiar to the people would be picked up. Evaluation requires knowledge of actual work, so evaluation professionals should be trained through the repetition of theory and practice.

(Talbot) Government bureaucrats know what is best, but their sense is sometimes different from people's sense.

(Kelman) As to human resources, people who have various skills and specialties should join the government sector. As to publicity, although accessing public information is possible, only limited people are using the information in the United States, too.

(Kanemoto) Policy evaluation is basically providing information, so people who are interested in the evaluation should read the information. In the case of PART, the conclusion comes first, so it is up to the people to read the details that follow. We have got to be ingenious so as to make the information attractive for readers. Public project evaluation in Japan is a similar case. A reason why the evaluation is not interesting is that projects that have not adopted are not evaluated. For each individual project, efforts should be made to make the information more dynamic, instead of only trying to increase its volume, and to make the contents more attractive. In order to carry out an effective discussion, evaluation should be made and opened much more in advance before the decision is made.


As to the development of professional human resources, we have a system to imploy middle career people or to retrain employees. However, since these systems are not working well, a more systematic approach is necessary. Since we have started to educate students using concrete policies of Japan, I hope future improvements can be expected.

(Questioner) Aren't there more intensive measures to make people interested?

(Niwa) The only thing we can do is discuss such policies that directly affect people's life and make patient efforts to follow the issues.

(Horie) Are there cases that the president or the prime minister shows interest in or refers to particular policy evaluations? If there are, what are some cases?

(Kelman) Prime Minister Blair has been holding meetings on the performance for the citizens. President Bush reviewed operation methods and improvement of management, rather than policies, in a more limited way

Closing Remark

Koichiro Agata, Associate Dean of the Okuma School of Public Management,
Waseda University

Feedback is important for policy evaluation as feedback to budgeting was stressed in today's discussion. However, it causes difficulties that the policy goal system and the budgetary system are not always consistent. This problem is not only in Japan. I think it is very important how three elements, which are the policy goal system, the budgetary system, and policy evaluation, are combined and feedback is done effectively.

For improving policy evaluation, I would like to request that universities and governments cooperate more closely. Universities can make contributions in at least two ways: improve the policy evaluation system by actually participating in the policy evaluation processes and contribute to human resource development. We would like to develop deserving human resources and improve government employee training.

We have built today an international network on policy evaluation. We would like to further improve that network and periodically exchange opinions. I sincerely hope that we will be able to contribute to the policy evaluation system in Japan and policy evaluation worldwide.