An international symposium titled "Importance of Policy Evaluation in the 21st Century and its Challenges: -Achieving further progress in Japan's policy evaluation system-" was held on June 25, 2006, by the Ministry of Internal Affairs and Communications (Administrative Evaluation Bureau), in conjunction with Waseda University (the Okuma School of Public Management). The symposium was held for the first time after the policy evaluation system was introduced.
|Opening Remark||Uichiro Niwa (Chairman of the Commission on Policy Evaluation and
Evaluation of Incorporated Administrative Agencies,
Chairman of ITOCHU Corporation)
|Keynote Speech||“Policy Evaluation in Japan
Problems and Solutions in six years period”
Michio Muramatsu (Professor, Faculty of Law, Gakushuin University)
|Special Speech 1||“Performance regimes and institutional contexts: comparing Japan,
UK and the USA”
Colin Talbot (Professor, The University of Manchester)
|Special Speech 2||“Evaluation and Performance Measurement: Tools to Improve
Government and /or to Shrink It?”
Steven Kelman (Professor, Harvard University)
|Welcome Address||Heizo Takenaka (Minister for Internal Affairs and Communications)|
|Panel-Discussion||“Importance of Policy Evaluation in the 21st Century and its Challenges
- Achieving further progress in Japan’s policy evaluation system-“
(Chairman of Subcommittee on Policy Evaluation of the
Commission of Policy Evaluation and Evaluation of
Incorporated Administrative Agencies, Professor,
University of Tokyo, Graduate School of Economics,
Graduate School of Public Policy)
Moderator: Masahiro Horie
(Vice-Minister for Policy Coordination, Ministry of
Internal Affairs and Communications)
|Closing Remark||Koichiro Agata (Associate Dean of the Okuma School of Public
Management, Waseda University)
[Summary of Speeches]
Uichiro Niwa, Chairman of the Commission on Policy Evaluation and Evaluation of Incorporated Administrative Agencies
Chairman, of ITOCHU Corporation
The basic philosophy of the reorganization of central government ministries is to convert postwar-type administration systems, which have been exhausted in many aspects during the 50 years after the war, to 21st century administration systems that would be more suitable for a free and fair society.
The policy evaluation system was newly introduced as a pillar to establish new administration systems. The new evaluation system was given three major points: accountability to the people, a high-quality efficient administration, and a results-oriented administration. The new system intends to create a scheme for reviewing policies and then reflect the result of the review on policy planning, which was not implemented effectively under the conventional administration programs.
About 10,000 policy evaluations have been conducted for the entire government of Japan, every year during the past four years. Furthermore, efforts have been made to reflect evaluation results on the budget request, which is to be submitted by the end of August of the previous year. Another successful outcome is that the percentage of the number of policies with numerical target level for performance evaluation has increased.
The committee will thoroughly implement the policy evaluation in line with key policies of the cabinet, and will strengthen the linkage of evaluations with budgeting and settling. In the meantime, as a future critical task, it is necessary to cope with the unavoidable problem that is the self-evaluation system, which lacks complete objectivity, and stress the importance of policy evaluation in such areas that concern people.
The committee should continuously watch the government's actions, and should submit messages when it is necessary.
Policy Evaluation in Japan - Problems and Solutions in six years period"
Michio Muramatsu, Professor at Gakushuin University
As the primary aspect of policy evaluation in Japan, self-evaluation is obligatory and its objectivity and strictness is ensured by the Ministry of Internal Affairs and Communications, and the evaluation method is flexibly left to each individual ministry. As the secondary aspect, all government personnel get involved in the evaluation, accepting evaluation as a valid part of government business. This is a very important part of this evaluation system. As the tertiary aspect, tension exists between sections in a ministry that mainly carry out policy activities and specialists in the same ministry who are engaged in policy evaluation. As the final aspect, the evaluation has been studied as a method, which is not bound by budgeting; hence the evaluation is not directly related to budgeting. This aspect is valuable in view of the nature of the evaluation.
There are three merits of Japanese policy evaluation. First, the legal scheme requires that all personnel be involved in the evaluation as a norm of government employees. Second, the introduction of policy evaluation has gives them the opportunity to strategically review the order of priority. Third, the evaluation considerably discloses to outside people what policies at individual ministry are valued and to what extent they are valued by these ministries. This has enhanced the government transparency.
At the same time, there are at least two issues to be addressed. First, it is not well seen how individual ministries utilize policy evaluation information and if they utilizes the information to efficiently manage to implement their duties. Second, there are extremely voluminous papers needed for policy evaluation. This implicates that it's not clarified what is to be done in the future in these papers.
It is said that the policy evaluation may have abolished some of projects. However, it does not seem that such policies, which were considered to be quite major but are somewhat questionable, were evaluated thoroughly. Now that policy evaluation is entering its second stage, evaluation should be taken one step further. Hopefully evaluations will be conducted thoroughly, and not as a hasty attempt to abolish some policies.
Special Speech 1
Today, I would like to compare Japanese reforms to similar reforms, which have taken place in the United Kingdom and the United States, and talk about the performance regimes and institutional context. The movement we have seen in the last 25 years toward more result reporting is not new. Today, most OECD countries are carrying out some sort of result reporting. As a recent trend, it is becoming larger, broader, deeper, and longer than any previous reform efforts in this field.
I would like to focus on three systems: Government Policy Evaluation Act in Japan, the U.K. Public Service Agreement (PSA) and the U.S. Government Performance and Results Act (GPRA). In Japan, "evaluation" as a term and as an approach is often used instead of "performance." On the other hand, in the United States and the United Kingdom, there are two large academic communities divided into those who promote policy evaluation and those who promote performance management. There is also a traditional difference between the two communities, which are divided vertically. Policy evaluation focuses on programs and policies, and evaluates individual policies, both qualitatively and quantitatively. In the meantime, performance evaluation focuses on organizations and continually evaluates plural programs with quantitative methods. However, in recent performance reporting, convergence of evaluation and performance has been observed in connection with the trend that long-term outcomes are have become increasingly valuable.
Looking at institutional factors, the process in the United Kingdom is quite simple; the Treasury negotiates public service agreements regarding performance targets with line ministries who in turn negotiate performance targets with their agencies. However, as a matter of fact, various factors exist to form a ministries' performance. In particular, Parliament gets involved by carrying out scrutiny and offering advice to government ministries about performance targets. The National Audit Office (NAO), courts, health bodies and education bodies, and users as the sixth factor may perform roles in forming performance. Finally, the last category is partner organizations.
I would like to look at the differences in the roles of core ministries and legislature among the United Kingdom, Japan, and the United States and compare these countries in terms of how both of them relate to line agencies. First, there are different systems in the United Kingdom, Japan and the United States in terms of the relationship between the executive and the legislative branches. In the United States, powers are separated, and there is a power contest between Congress and the executive branch. The United Kingdom has an executive-dominated system, and Japan leans slightly further toward the separation of powers. While GPRA comes from the legislature in the United States, it comes from the executive in Japan and the United Kingdom. This difference is not surprising if the roles of the executive and the legislative branches in these three countries are considered.
Let's examine how result reporting is integrated into the budget. The United Kingdom claims that performance information is integrated into the budgetary process. However, there is no evidence that performance results are actually used in the budgetary process. Although the United States is listed at a high level of performance integration into the budgetary process, this is because the United States claims so, yet there is no clear evidence.
There are variations in how much reporting is done. While the Treasury negotiates PSA performance targets with line ministries in the United Kingdom, line ministries set up their performance targets in Japan and the United States. In the latter two countries, line ministries have great ownership of these processes.
Ministries are disclosing to the outside such information showing how they utilize performance results effectively in order to improve their performance. However, these ministries are utilizing little of the information within their organization.
Special Speech 2
Policy Evaluation: Tools to Improve Government and/or Shrink It?
Steven Kelma, Professor at Harvard University
The goal of policy evaluation is to change the policymaking process, i.e. to try to add a new element to the traditional policymaking process that has been dominated by political parties, interest groups, public opinion, and the like. The new element to be added is facts and evidence.
As to the roles of Ministry of Internal Affairs and Communications, first, the ministry should support development of tools for performance measurement and collect best practices of performance improvement. Second, the ministry should support development of research methods and techniques in the private sector, particularly at universities for doing evaluation studies. Lastly, Ministry of Internal Affairs and Communications should collaborate with Ministry of Finance by giving performance information and analyzing performance information to assist Ministry of Finance in using the information in the budget process.
Minister of Internal Affairs and Communications
Policy evaluation, which is extremely important, just gets understandings among the public. However, it should be more well-known to the people. In democratic societies it is the singl
e greatest way for enhancing policymaking performance to let the people know thoroughly the meanings of policies and policy evaluations that are going on.
Since 2001 in Japan, the policymaking process has changed drastically. First of all, the process of determining the basic direction of policy has become established as a "Big-boned policies", which are formulated after discussing policy intensively and thoroughly. The procedure has explicitly changed to such a process where a basic policy line is decided in June and thereafter budgeting is studied until December. In this innovative change, the foundation for evaluating and discussing policies orderly is being set up in Japan. Although policy evaluation is taking its steps forward these days in Japan, it has not yet been rooted sufficiently.
All the people attending this symposium today are highly knowledge and deeply interested in policy evaluation. I hope you will make discussions in a focused manner and communicate extensively to the public. There are many challenges that the Ministry of Internal Affairs and Communications should tackle. I hope you will give us suggestions, including strenuous ones.
"Importance and Challenges of Policy Evaluation in the 21st Century: For Development of Policy Evaluation in Japan"
○Panelists: Uichiro Niwa
Yoshitsugu Kanemoto, Chairman of Subcommittee on Policy Evaluation of
Commission of Policy Evaluation and Evaluation of
Incorporated Administrative Agencies, Professor, University
of Tokyo, Graduate School of Economics, Graduated School
of Public Policy
<Challenges of policy evaluation in Japan>
(Kanemoto) One feature of policy evaluation in Japan is that all the three types of evaluation; project evaluation, performance evaluation, and comprehensive evaluation, have been introduced into all the ministries, simultaneously under the Government Policy Evaluation Act. However, the comprehensive evaluation, which corresponds to program evaluation, has not been developed significantly.
There are such problems that, first, persons who evaluate policies lack high morale and, second, policy evaluation is low in the professional analysis level.
(Talbot) For improving policy evaluation in Japan, the volume of reporting should be reviewed, and independent validation should be established.
(Niwa) I think the introduction of the policy evaluation system was extremely meaningful. Third-party evaluation would be difficult without a lot of professional personnel. External evaluation is meaningless if it is the same as self-evaluation in contents. If a secondary evaluation is made by involving a third party, then a new perspective could be brought in, such as quantification of target level of their policies. As for making comprehensible evaluation reports, it should be selective and concentrative what are included in.
(Kanemoto) Difficulty in improvement is in relation to the self-evaluation system adopted in Japan. Another problem is that the Ministry of Finance is in no position to get into the process of evaluation. Nevertheless, I think improvements are being made.
(Kanemoto) Policy evaluation is basically providing information, so people who are interested in the evaluation should read the information. In the case of PART, the conclusion comes first, so it is up to the people to read the details that follow. We have got to be ingenious so as to make the information attractive for readers. Public project evaluation in Japan is a similar case. A reason why the evaluation is not interesting is that projects that have not adopted are not evaluated. For each individual project, efforts should be made to make the information more dynamic, instead of only trying to increase its volume, and to make the contents more attractive. In order to carry out an effective discussion, evaluation should be made and opened much more in advance before the decision is made.
(Horie) Are there cases that the president or the prime minister shows interest in or refers to particular policy evaluations? If there are, what are some cases?
Koichiro Agata, Associate Dean of the Okuma School of Public Management,
Feedback is important for policy evaluation as feedback to budgeting was stressed in today's discussion. However, it causes difficulties that the policy goal system and the budgetary system are not always consistent. This problem is not only in Japan. I think it is very important how three elements, which are the policy goal system, the budgetary system, and policy evaluation, are combined and feedback is done effectively.
For improving policy evaluation, I would like to request that universities and governments cooperate more closely. Universities can make contributions in at least two ways: improve the policy evaluation system by actually participating in the policy evaluation processes and contribute to human resource development. We would like to develop deserving human resources and improve government employee training.
We have built today an international network on policy evaluation. We would like to further improve that network and periodically exchange opinions. I sincerely hope that we will be able to contribute to the policy evaluation system in Japan and policy evaluation worldwide.