|
TOWARDS
THE INTERNATIONAL EVALUATION
Harry Hillman Chartrand © |
|||||||||||||||||||||||||||||||||
Evaluation is a term with many meanings to different people. Among social scientists from different disciplines its meaning and practice varies. Between artists and social scientists its meaning serves as a lightening rod for conflict and controversy. Accordingly, evaluation of arts council funding must account not only for nuances in scientific methodology, but also for a fundamental dissonance between rational ways of knowing such as scientific evidence, and arational ways of knowing such as artistic judgment (Chartrand, 1979). The challenge of reconciling scientific evidence and artistic judgment has socio-political implications extending far beyond evaluation of arts council funding (Bell, 1976). Such a reconciliation would contribute towards actualizing the potential for human learning, which has been called the ultimate planetary resource (Botkin, Elmandirjra, Malitza, 1979); towards developing a more mature, balanced and healthy citizenry (McLuhan, 1978); and, towards defining goals and values for public policy appropriate in the post-modern era (Harman, 1979). Before examining a proposed framework for evaluating arts council funding, it is appropriate to define what is an arts council, and explain why an evaluation framework is necessary at this time. In theory, there are four alternative roles for government in support to the arts: Facilitator, Architect, Engineer, and Patron (Chartrand, McCaughey, 1985). In the real world, most governments play some mix of all four roles. The Facilitator funds the arts through tax expenditures made on behalf of, and according to, the tastes of individual, corporate, and foundation donors. The United States is the best example of a Facilitator State. The Architect funds the arts through ministries or departments of culture generally as part of social welfare policy, and promotes community standards. Western European countries are examples of Architect States. The Engineer owns the artistic means of production, and uses them to promote political education. Eastern bloc countries are examples of Engineer States with respect to arts funding. The Patron funds the arts through arm's length councils consisting of a Board of Directors that operates like the management of a blind trust, in this case to promote standards of professional excellence. The council receives funds from the government, but government does not determine which artists or arts organizations will receive support. Arm's length arts councils are the principal vehicle for public support to the arts in English- speaking countries. In Western countries, the arm's length principle is applied in a wide range of constitutional and public affairs such as separation of powers between executive, legislative, and judicial branches of government, division of powers among orders of government in federated states, and application of human rights legislation through an independent ombudsperson. Essentially, the arm's length principle forms the basis for the system of "checks and balances" used in pluralistic democracies to avoid undue concentration of power and conflict of interest. Its use in arts funding is not therefore exceptional, but rather an example of a general principle applied to a specific public policy sector. There are three reasons why it is critical at this time to develop an effective framework for evaluating arts council funding. First, over the past decade the arts increasingly have become an important issue on the public policy agenda (Gulbenkian, 1981, 26). This reflects on the one hand changing demographics such as rising levels of education, the increasing role of women in economic and political life, and the aging of the general population which together have resulted in growing participation in arts-related activities (Chartrand, 1984). On the other hand, it reflects the growing economic importance of the arts implicit in their transformation from symbol to source of national wealth (Chartrand, 1986). Second, public support to arm's length arts councils, measured in constant dollars, has remained constant or declined in the past ten years. This has resulted in heated competition for increasingly scarce funding between national "flagship" institutions and emerging artistic enterprise. Part of this controversy is reflected in the issue of "double-arm's length", i.e., an arts council, having assessed the artistic merit of clients, should not direct or control their activities (Wearing, 1983, 13). Third, in Western countries rising standards of accountability are now being applied to all public sector activities. In future, the price of autonomy from government direction currently enjoyed by arm's length arts councils may well be standards of accountability higher than those applied to other areas of public activity. Such higher standards can be called "transparency to public scrutiny". While arts councils have been judged successful in harnessing artistic judgment (Applebaum, Hebert, 1982, 32; Education, Science and Arts Committee, 1982, xlvi; Wyszomirski, 1983, 17; Macaulay, 1984), they have been unable or unwilling to integrate scientific evidence into their decision making processes (Nissel, 1979; Urice, 1983). Scientific evidence used in evaluation of arts council funding, however, must be subjected to artistic judgment as to its relevance and meaningfulness. It can serve to enhance the "informed intuition" (Jantsch, 1967) of decision makers, and thereby amplify the quality of their artistic judgment. It cannot, and should not, however, be considered as a substitute or alternative to such judgment.
To
conduct an international evaluation of arts council funding it is necessary to
describe arts council objectives, with particular attention to issues which can
serve as the basis for trans-national comparison (Figure 1). It is also
necessary to define appropriate performance measurement program goals and
criteria. Using the proposed framework a series of a priori hypotheses will be
derived. To test these hypotheses an extensive research program is
required including creation of various monitoring instruments. The most
important of these instruments are citation and issues indices (C&I Indices)
of the record of arts council debate and decision cross-referenced to all of the
issues, goals, and criteria defined below. Figure 1 FRAMEWORK
OF EVALUATION
Evaluation begins with definition of stated objectives. In the case of arm's length arts councils this is difficult for four reasons. First, as suggested in the introduction, there is an inherent reluctance on the part of arts councils and the artistic community to define objectives in rational and unambiguous terms amenable to traditional evaluation methodologies (Hendon, 1980, 367). Second, the legislative mandate of arm's length councils, as will be demonstrated, is enabling, not operational in nature. Third, the operating objectives of arts councils tend to be dynamic, changing through time in response to the growth and maturation of the artistic community, e.g., the emergence of new arts forms like video. Fourth, arts councils, as public policy agencies (Breton, 1977), tend to have a hierarchy of objectives that must be defined at a mandate, operating, policy, program, and procedural level.
The legislative mandate of an arm's length arts council is, by definition, enabling rather than operational in nature, i.e., legislation sets out general powers leaving to the council interpretation and definition of these powers. In the case of Canada Council, for example, the Act states that the Council should "foster and promote the study, the enjoyment, and production of works in the arts". The Act provides examples of art forms, and means by which they may be supported "without limiting the generality of the foregoing" (Canada Council Act, 1957). The enabling nature of legislation means that the operating objectives, policies, programs and procedures of arts councils are self-determined.
There
are three issues, however, which should permit comparative evaluation of the
mandates, and resulting funding patterns of different arts councils. The
first is whether a council is an agent of the executive, or the legislative
branch of government. The second is whether final granting authority rests
with the Chairperson, or with the Board of Directors. The third is whether
the legislative mandate places explicit emphasis on consumption of the arts,
i.e., audiences. Similarly, if legislative emphasis is placed on consumption, then the pattern of funding will tend to reflect a high priority for arts education in the schools, audience development, community arts, and outreach activities, such as television and other media programming about the arts. Moreover, members of the Board and staff will tend to be more representative of the general public than of artistic constituencies. If no legislative emphasis is placed on consumption, then the funding pattern should be more responsive to the creative and experimental aspirations of the artistic community, than to the usually more conservative tastes of the general public. In this case, an arts council will tend to become, like regulatory agencies, captive of the regulated sector, i.e., of the artistic constituencies. Board members and staff will be selected for their knowledge and practice of the arts rather than expertise in management, or representativeness of the general public. To test hypotheses concerning arts council mandates, four research projects are required. First, formal content analysis of legislative instruments is required to determine which are executive and which are legislative agencies, whether final authority resides with the Chairperson or the Board, and whether explicit legislative emphasis is placed on consumption. Second, citation and issue indices of debate by the legislative and executive branches of government concerning the arts are required to provide a basis for determining council responsiveness to changing government policy priorities. Such indices should be cross-referenced to the program goals described below. Third, socio-demographic profiles of Board members and staff are required to determine whether they are representative of artistic constituencies or the general public (Harris, June 1969). Fourth, calculation of the relative share of council grant-giving to consumption activities, such as audience development and education, is required.
Operating Objectives The operating objectives of an arm's length arts council emerge through interpretation of the legislative mandate by successive Boards and Chairpersons. Essentially, operating objectives concern what types of art, including emerging art forms, are supported. There are two issues which should permit comparative evaluation of the operating objectives and the resulting pattern of funding of arts councils. The first is the balance between support to the fine, the commercial, and the amateur arts (Chartrand, 1984). This balance is important, because collectively the three types constitute the arts industry, one of the fastest growing and largest employers in Western economies. The three are intimately related in that the individual artist is the ultimate source of art in all three. Furthermore, the amateur arts, in actualizing the talents and abilities of the individual citizen, also provide an educated audience and initial training for the fine and the commercial arts. The fine arts, in the pursuit of artistic excellence as an end in itself, provide research and development for the commercial arts. The commercial arts, in the pursuit of profit, provide the means to market and distribute the best of the amateur and the fine arts to a large enough audience and in a form suited to earning a profit. It is the commercial arts which have become, over the past decade, an increasingly important national economic policy issue, with a resulting reduction in the policy priority assigned to the fine arts. While arts council funding is generally directed to support the fine arts, and to a lesser degree the amateur or community arts (Gulbenkian, 1981, 43), arts councils also provide some direct or indirect support to the commercial arts. This may take the form of venture capital grant programs to encourage development of commercial theatre, or script development grants in film. A Canadian example involves a script development grant awarded by the Media Arts Section of the Canada Council that led to production of the commercially successful motion picture, My American Cousin. The second issue permitting comparative evaluation of operating objectives is the range of artistic disciplines and sub-disciplines supported, as well as responsiveness to new and emerging art forms. Arts councils have tended to support traditional art forms in dance, music, opera, theatre, visual arts, and literary writing. Emerging sub-disciplines arise from time to time, however, that test the quality of an arts council's artistic judgment in recognizing new arts forms in a timely manner, and its commitment to support such new forms. For example, modern and experimental dance have emerged as significant new art forms in the past decade, as have media art forms such as film, video, and integrated technology. To conduct a comparative evaluation of the operating objectives of different arts councils, three steps are necessary. First, citation and issue indices of the record of decision are required to determine stated operating objectives. Second, comparative financial analysis is required of support to the amateur, the commercial, and the fine arts, including traditional disciplines and emerging art forms. Third, comparison between stated objectives and actual results is required.
Arts council policy can be characterized as strategy, i.e., the overall manner in which resources are organized so as to achieve operating objectives. All arm's length arts councils have adopted "excellence" as the strategic principle to guide funding. Excellence, however, is a relative term which involves the exercise of artistic judgment at various levels of artistic activity. There are two issues which should permit comparative evaluation of the policy objectives, and the resulting pattern of funding of arts councils. The first is whether an arts council defines standards of excellence at the national, regional, or local level. The second issue is the financial balance between support to artistic activities in major metropolitan areas versus what can be called hinterland areas. If arts council funding is guided by excellence defined at the national level, then it is likely that the majority of support impacts in major metropolitan areas. Such support will be directed at existing, established institutions. If, on the other hand, funding is guided by excellence defined at the regional or local level, then support will tend to be developmental in nature fostering emerging artistic enterprises and impact in the hinterland areas. To test hypotheses concerning arts council policy objectives, three steps are required. First, content analysis of the record of decision is required to determine if excellence is defined at the national, regional, or local level. Second, comparative financial analysis of support to established versus emerging artistic enterprise and to major metropolitan versus hinterland areas is required (Hutchinson, 1982; Research & Evaluation, 1985). Third, comparison between stated policy objectives and financial results is required.
Arts council programs can be characterized as tactics, i.e., reduction of strategy into clearly defined operational means by which objectives can be achieved. In effect, the tactic of arts councils is to organize support into programs in aid to individual artists and to arts organizations. Accordingly, a primary issue in evaluation of arts council funding is the balance between support to individual artists and arts organizations Two secondary issues arise. First, in the case of individual artists the issue is the balance between support in the form of prizes recognizing attained excellence, and grants to further career development. Second, in the case of arts organizations what is at issue is the balance between support to large "flagship" institutions and small, emerging arts companies. Beyond these general questions, it is also necessary to develop clearly defined program goals and criteria for measuring the degree to which arts councils have attained their program objectives. Unfortunately, there is no generally accepted set of program goals or criteria in arts policy (Schuster, 1975). After a review of the literature and the activities of the Canada Council, 7 inter-related program goals for arts council funding of each art form can be identified. They are: TRAINING of artists, arts technicians, and arts administrators; CREATION of new art works; PRODUCTION of art works by organizations, as well as employment of the arts-related work force in production facilities such as publishing houses, performing arts facilities and visual arts galleries; DISTRIBUTION of works of art including publishing, exhibitions, touring and media extension of the performing and visual arts; CONSUMPTION of works of art including the study of the nature and composition of the arts audience; COMMUNICATIONS within the various arts communities, including representative associations, unions, trade journals, conferences and seminars; and CONSERVATION of art works, including maintaining classic works in the performing repertoire, as well as preservation of works in the literary, media and visual arts. Each arts council granting program can be classified according to one, or more, of these goals. Each goal applies not just at the disciplinary level but also at the sub-disciplinary level, e.g., ballet vs. modern dance. Each program should be linked with three sets of measurable performance criteria. First, procedural efficiency criteria are required in the form of input indicators of unit cost in financial and human resources associated with grants and services. Indicators would include peer evaluation and officer cost per application, cost per juror and officer day, cost per jury meeting, composition of juries by sex, region and language, candidate costs per application including audition or art work expenses. Attained levels and trends in input indicators should be assessed against targets set by the Board. Second, program effectiveness criteria are required in the form of output indicators such as the number and distribution of grants by region, sex and language, constant dollar value, average constant dollar value, annual and average annual rate of change in grant support. Attained levels and trends in output indicators should also be evaluated against targets set by the Board. Third, policy impact criteria are required in the form of indicators of support provided relative to need, as well as cost comparisons with alternative methods of delivery. Indicators would include activity supported relative to total activity, e.g., grantee employment rates versus unsuccessful candidates, new works produced through art council support versus total number of new works, productions distributed through tours, spectators per performance, and spectators per dollar of support. Quantitative indicators must be subjected, however, to artistic judgment. For example, a high spectator per performance indicator in 19th century symphonic music should not be compared with a low one in experimental electronic music. Artistic judgment is required if electronic music, a new and developing art form is to be distinguished from symphonic music, an established form. If electronic music is judged to be artistically important, and if it is to mature, then it must be supported in the short term, even if few spectators attend. To test hypotheses concerning the program objectives of arts councils, four steps are required. First, citation and issue indices of Board decisions concerning program goals and criteria must be constructed. Second, comparative financial analysis must be undertaken concerning support to individual artists and arts organizations based upon trend data by program type including administrative costs, preferably in the form of standard objects of expenditure (Research & Evaluation, May 1984). Third, further definition and application of proposed performance measurement goals and criteria is required. Fourth, comparison of stated program goals and target criteria levels and actual results is required.
Arts council procedures can be characterized as logistics, i.e., the actual application of resources to achieve objectives and policies through individual programs. Arts council procedures are generally based on systems of peer evaluation. A primary policy issue permitting evaluation of arts council funding is the balance between the use of individual assessors, jurors, or officers in awarding grants. A secondary issue is the degree to which arts council procedures are transparent or opaque to public scrutiny. The jury system is generally held up by arts councils to government and the public as a guarantee of impartiality in awarding grants. Individual assessors, however, provide confidential reports not available to the general public. Similarly, if officers award grants without independent assessment or a formal jury, council procedures are also hidden from public scrutiny. If the majority of arts council funding is awarded by assessors and officers then procedural impartiality is open to question. Furthermore, the use of the same jurors from the same sub-disciplinary background can, over time, bias arts council funding towards some, and away from other sub-disciplines. Without empirical evidence concerning the frequency with which assessors and jurors are employed, it is a moot question whether given sub-disciplines, individual artists, and arts organizations receive impartial peer evaluation. Artistic judgment must be used to assess the relevance and meaningfulness of resulting quantitative and descriptive indicators. For example, the relevance of language and culture to the definition of a peer in the assessment of excellence is an issue which serves to highlight the importance of artistic judgment. Some observers in Canada, for example, argue that jurors from the English-speaking dance tradition can not assess excellence in dance derived from the French tradition. Others argue that excellence in dance transcends all cultural and linguistic traditions. Accordingly, definition of a peer is more a question of artistic judgment than simple socio-demographic data such as years of education or training. To test hypotheses concerning arts council procedures, four steps are required. First, content analysis of the record of decision concerning Board priorities for the use of assessors, jurors, and officers is required. Second, comparative financial analysis of support awarded by assessors, jurors, and officers is required based on trend data by program type including administrative costs. Third, a peer evaluation profile of assessors, jurors, and officers is required including socio-demographic data, frequency of employment, history of grants to assessors and jurors before and after employment in the peer evaluation system, as well as professional resumes documenting the sub-disciplinary affiliation of assessors, jurors, and officers. Fourth, evaluation of attainment of stated procedural objectives relative to actual results is required.
Development of a framework for the evaluation of arts council funding is important because of the increasing public policy profile of the arts, declining real resources available to arts councils resulting in increased competition between established and emerging artistic enterprise, and rising standards of public accountability which have special relevance for maintenance of the autonomous arm's length status of arts councils. Evaluation of arts council funding requires, however, reconciliation of scientific evidence and artistic judgment. The challenge of integrating these dissonant ways of knowing extends far beyond evaluation of arts council funding to actualizing human learning potential, developing a more mature, balanced and healthy citizenry, and defining public policy objectives appropriate in the post-modern era. Accordingly, scientific evidence used in evaluation of arts council funding must be subjected to artistic judgment as to its relevance and meaningfulness. It can serve to enhance the "informed intuition" of decision makers, and thereby amplify the quality of their artistic judgment. It cannot, and should not be considered as a substitute for such judgment. Given the autonomous status of arm's length arts councils, as well as the complexity of the arts as a public policy sector, it is disappointing how little research has been conducted concerning the efficiency, effectiveness, and impact of arts council activities. As indicated by a priori hypotheses derived from the proposed evaluation framework, there are significant questions about the operation of arm's length arts councils throughout the world. It is hoped that this discussion will encourage researchers and arts council decision makers, to undertake the long, difficult process required to develop a more full and complete understanding of the international "arts council phenomenon" (Sweeting, 1982). Applebaum, L., J. Hebert, Report of the Federal Cultural Policy Review Committee, Minister of Supply and Services, Ottawa, 1982. Bell, D., The Cultural Contradictions of Capitalism, Basic Books, NY, 1976. Botkin, J., M. Elmandjra, M. Malitza, No Limits to Learning: Bridging the Human Gap - A Report to the Club of Rome, Pergamon, Toronto, 1979. Breton, R., The Canadian Condition: A Guide to Research in Public Policy, Institute for Research on Public Policy, Montreal, 1977. Chartrand, H.H., The Canadian Cultural Industries, Futures, Ottawa, 1979. Chartrand, H.H., Social Sciences & Humanities Research Impact Indicators, Futures, Ottawa, January 1980. Chartrand, H.H., An Economic Impact Assessment of the Canadian Fine Arts, Third International Conference on Cultural Economics & Planning, Akron, Ohio, April 1984.
Chartrand,
H.H., "The Value of Economic Reasoning and the Arts", in The Chartrand, H.H., C. McCaughey, The Arm's Length Principle and the Arts: An International Perspective - Past, Present, and Future, Research & Evaluation, Canada Council, Ottawa, September 1985. Cwi, D., Arts Councils as Public Agencies: The Policy Impact of Mission, Role and Operations, Second International Conference on Cultural Economics and Planning, Maastricht, Netherlands, May 1982. Education, Science and Arts Committee, Public and Private Funding of the Arts, Eighth Report, House of Commons, Her Majesty's Stationery Office, London, England, 1982. Harman, W., "Mind Research & Human Potential", Congressional Clearing House on the Future, Vol. 4, No. 5, November 1979. Harris, J., "Decision-Making in Government Programs of Art Patronage: The Arts Council of Great Britain", Western Political Quarterly, June 1969. Hendon, W., "An Observation on Arts and Economics", in Economic Policy for the Arts, W. Hendon, J. L. Shanahan, A. J. MacDonald (ed), Abt Books, Cambridge, Massachusetts, 1980.
Hutchinson,
R., A Hard Fact to Swallow, Policy Studies Institute, London, Jantsch, E., Technological Forcasting in Perspective, OECD, Paris, 1967. Jantsch, E., Design for Evolution, Braziller, NY, 1975. Macaulay, R.W., P. Day, G. Sherman, Report to the Honourable Susan Fish, Minister of Citizenship and Culture by the Special Committee for the Arts, Toronto, Spring 1984. McLuhan, M. "The Eye and the Ear and the Hemispheres of the Brain", Futures Canada, Vol. 2, No. 4, 1978. Nissel, M., The Statistical Needs of the Arts Council, Policy Studies Institute, London, United Kingdom, July 1979. Research & Evaluation, Canada Council System of Standard Objects of Expenditure, Canada Council, May 1982. Research & Evaluation, Canada Council Provincial and Metropolitan Accounts 1971 to 1984, Canada Council, September 1985. Schuster, J.M., Program Evaluation and Cultural Policy, Department of Urban Studies and Planning, MIT, September 1975. Sweeting, E. (ed), Patron or Paymaster? The Arts Council Phenomenon, Report on the Second Conference of Commonwealth Arts Councils, Calouste Gulbenkian Foundation, London, England, 1982. Urice, J.K., "Using Research to Determine, Challenge, or Validate Public Arts Policy", Journal of Arts Management and Law, Spring, 1983. Urice, J.K., Federal Arts Policy and Research: The Non-Relationship, Tenth Annual Conference on Social Theory and the Arts, Rutgers University, New Brunswick, New Jersey, October 22, 1983. Wearing, Joseph, Funding the Arts: A Cross National Perspective, 25th Annual Conference of the Western Social Science Association, Albuquerque, New Mexico, 29 April 1983. Wyszomirski, M., The Reagan Administration and the Arts: 1981-1983, American Political Science Association, Chicago, Illinois, September 1-4, 1983.
|
|||||||||||||||||||||||||||||||||