Qual Quant (2013) 47:2883–2913 DOI 10.1007/s11135-012-9696-1

Accreditation and standards-driven program evaluation: implications for program quality assurance and stakeholder professional development Saad F. Shawer

Published online: 16 March 2012 © Springer Science+Business Media B.V. 2012

Abstract This study evaluates a language education program and assesses the influence of this program evaluation on program performance and stakeholder professional development. The research design makes use of mixed paradigms and methods. Positivism is followed to verify the extent to which the program meets predetermined quality standards through quantitative evaluation research, questionnaires and document and record analysis. In contrast, the qualitative paradigm is followed to uncover program evaluation impact on program components and stakeholders through qualitative evaluation and interviews. The study concludes the program maintains satisfactory overall performance with some components performing better than others. Likewise, evaluation of program evaluation improves program elements and faculty and program administration professional skills. The study therefore recommends program evaluation as an effective program improvement strategy. Program evaluation is also an effective professional development strategy that provides program administration, staff and faculty members with onsite opportunities to develop their professional skills. Moreover, evaluation of program evaluation should be an integral part of program evaluation to help program stakeholders not only do but also use evaluation alongside decreasing resistance to imposed review and reform. Keywords Language program · Language learning · Quality assurance · Program evaluation · Accreditation · Standards · Professional development

1 Introduction For several decades, governments worldwide have been concerned about the quality and relevance of higher education (Sywelem and Witte 2009). Recent economic uncertainties

S. F. Shawer (B) Department of Teacher Education, Arabic Language Institute, King Saud University, P.O. Box 4274, Riyadh 11491, Kingdom of Saudi Arabia e-mail: [email protected]; [email protected]; [email protected]; [email protected]

123

2884

S. F. Shawer

have even increased political pressures on higher education institutions to show their programs meet national and international accreditation standards. Higher education institutions in turn impose program evaluation to address accreditation demands and to demonstrate their programs deliver what they promise (Byrnes 2006; Norris 2009). Like many higher education institutions, King Saud University demanded all colleges to show their programs meet national and international standards. In June 2009, dismaying news was communicated that all programs are on probation and unless they satisfy the National Commission for Academic Accreditation and Assessment (NCAAA) standards by September 2012, they would shut down. As a result, this article reports on a two-year self-evaluation of a language education program to address accreditation standards. In this four-semester program, students study 32 courses of 80 credits. Program courses address the four skills (listening, reading, speaking and writing) alongside the target language culture and grammar. The program grants full scholarships to international students who meet admission criteria and students who successfully complete the program receive a Proficiency in Language Diploma. In this context, this study sought to evaluate this program and assess its influence on program performance and stakeholder professional skills.

2 Program evaluation A program is an intervention introduced to achieve external objectives so that some recognized need or problem could be addressed, whereas program evaluation concerns “an informationgathering and interpreting endeavor that attempts to answer a specified set of questions about a program’s performance and effectiveness” (Rossi et al. 1999, p. 62). Although evaluation involves formal and informal judgments about program value, formal evaluation “applies scientific procedures to the collection and analysis of information about the content, structure and outcomes of programs” (Clarke 1999, p. 1). The cornerstone in program evaluation is the use of resulting information to improve, change or terminate programs (Stake 2011). In this respect, quality of program objectives and outputs has paramount importance to program evaluation; since program objectives are the formal goals to which program resources are directed while program outputs are the services programs provide (Shawer 2012). Now that “programs exist in order to change, enrich, enhance, extend, or improve the lives of participants and, by extension, the quality of life in society as a whole,” governments and the public have the right to ascertain that programs deliver what they promise through standards-driven program evaluation (Norris 2006, p. 577). Program evaluation provides program stakeholders with an opportunity to show clientele, institutions and society who they are, why they exist and what services they deliver. In the same vein, it answers accrediting bodies why they should not be shut down (Norris 2006). A language education program “generally consists of a slate of courses designed to prepare students for some language-related endeavor” (Lynch 1996, p. 2). To know what works in language programs, program evaluation is needed to generate important information about language proficiency gains (Ross 2003). Program evaluation therefore is essential not only to improve program performance (planning, design, implementation and outcomes) but also to meet imposed institutional requirements (Lynch 1996). Although program evaluation has been the main mechanism for programs to demonstrate they address quality, public accountability and accreditation concerns, imposed program evaluation was considered by program stakeholders as threats rather than improvement opportunities. This puts program evaluation utility in jeopardy because stakeholders undertake program evaluation as an end rather than a means for improvement. Effective program

123

Accreditation and standards-driven program evaluation

2885

evaluation in general and language program evaluation in particular can be a powerful improvement strategy when stakeholders and clientele focus on using more than doing evaluation (Norris 2006).

3 Accreditation and standards Governments use accreditation to ensure higher education addresses quality and relevance concerns. Likewise, institutions use program evaluation to meet higher education accreditation standards (Sywelem and Witte 2009). Accreditation therefore is a quality assurance procedure to maintain and improve quality through measuring to what extent higher education institutions meet standards of good quality. Moreover, “accreditation passes a verdict on whether programs, degrees or institutions meet certain outside standards or requirements” (European Network for Quality Assurance in Higher Education 2001, p. 7). As such, accreditation is “a process of external quality review used by higher education to scrutinize colleges, universities, and educational programs for quality assurance and quality improvement” (Council for Higher Education Accreditation 2002, p. 1). Accreditation helps students choose colleges, protects institutions against harmful pressure and helps them create self-improvement goals alongside setting professional certification and licensure criteria. Standards have become a prerequisite to national and worldwide recognition of institutions that they meet quality criteria on grounds of consistent and rigorous examinations. They address society demands by increasing the reliability and effectiveness of processes and services. A standard is an agreed and repeatable procedure that sets out the criteria of doing something (British National Standards Body 2011). As such, standards are essential to accreditation and program evaluation (Sywelem and Witte 2009), allowing programs to compete on equal terms through, for example, the International Organization for Standardization (ISO) which grants worldwide recognition to institutions, programs or services that meet international standards. Standards improve programs through supporting innovation, promoting productivity and developing and maintaining best practice. Institutions or products earning certification marks assure customers that recognized institutions consistently stood up to rigorous examination. Moreover, institutions benefit from the conformity and integrity that standards offer through developing and adopting best practice (British National Standards Body 2011).

4 Professional development and program evaluation Since professional development (PD) includes “all types of professional learning …beyond the point of initial training” (Craft 1996, p. 6), learning through program evaluation is a form of PD. Program evaluation provides optimal opportunities not only for one-shot PD but also continued professional development that involves “those ongoing formal and informal learning activities through which professionals continue to advance their professional competence so that they can improve their practices and profession” (Shawer 2010, p. 598). This way program evaluation creates a context where stakeholders continue to improve professional knowledge and skills during their careers. Such learning opportunities concur with current trends that take professional development as a career-long process (Anderson and Olsen 2006). Despite the plethora of research on generic-education and language program evaluation, there was disparity among prior research findings. A category of studies assessed program per-

123

2886

S. F. Shawer

formance (how does a program or program component work?). Many of these studies assessed program learning and teaching and found satisfactory language proficiency and good instruction (Arnold 2009; Bernhardt 2006; Houston 2005; Rivera and Matsuzawa 2007; Topkaya and Kucuk 2010) but others showed unsatisfactory performance (Dassier and Powell 2001; Gorsuch 2009; Sanders 2005). Likewise, other studies assessed program goals and objectives and indicated satisfactory performance (Arnold 2009; Middlebrook 1991; Topkaya and Kucuk 2010) while others did not (Dassier and Powell 2001; Rivera and Matsuzawa 2007). Invariably, some other studies found satisfactory program resources in terms of library, resource centers, ratio of faculty to students, staff, equipment and buildings (Arnold 2009; Elder 2009; Kiely and Rea-Dickins 2005; Middlebrook 1991) but other studies showed poor performance (Topkaya and Kucuk 2010). Some studies also found satisfactory program administration (Chase 2006; Kiely and Rea-Dickins 2005; Middlebrook 1991), employment processes (Middlebrook 1991), relationships with the community (Pawan and Thomalla 2006) and student support services, including admissions, orientation, advising, financial aid and housing (Dassier and Powell 2001; Middlebrook 1991; Topkaya and Kucuk 2010). Moreover, several studies found satisfactory program development opportunities (Bernhardt 2006; Dassier and Powell 2001; Kiely 2006; Norris 2009) while other studies did not (Topkaya and Kucuk 2010). A second category of studies evaluated the influence of program evaluation on program improvement (what use did a program make of program evaluation results?). This category of research marks a move from doing to using evaluation and has been dramatically increasing at the turn of the 21st century. These studies found program evaluation, whether internally-motivated or externally imposed, resulted in significant improvement in the language education program, including, for example, program goals and objectives, curriculum and instruction, learning outcomes and use of resources (Carsten-Wickham 2008; Dassier and Powell 2001; Kiely 2009; Norris 2009). The third research category evaluated the influence of program evaluation on the professional development of program stakeholders (what impact did program evaluation have on program stakeholders?). Not only has the number of these studies increased at the turn of this century, but also marks a move toward using program evaluation as a professional development strategy. For example, faculty members developed research skills, student assessment skills and became aware of program elements (Carsten-Wickham 2008; Chase 2006; Dassier and Powell 2001; Kiely 2006; Norris 2009; Sullivan 2006). In the light of this review, the present study sought to answer these research questions: 1. 2. 3.

To what extent did the program meet the 11 standards of quality performance? What use did the program make of program evaluation? How did program evaluation impact on program stakeholders?

5 Research design As shown in Fig. 1, positivism guided this research through verifying the quality of predetermined variables (Cohen et al. 2011). Quantitative evaluation was therefore used to address the first research question through the collection, analysis and interpretation of numerical data about program effectiveness against 11 NCAAA standards in order to use the resulting information for program improvement (Clarke 1999; Shawer 2012). The NCAAA standards include program (1) mission, goals and objectives, (2) governance and administration, (3) management of quality assurance and improvement, (4) teaching and learning, (5) student administration and support services, (6) learning resources, (7) facilities and equipment, (8)

123

Accreditation and standards-driven program evaluation

2887

Fig. 1 Research design

financial planning and management, (9) employment, (10) research, and (11) relationships with the community. Figure 1 also shows the qualitative paradigm was needed to uncover and understand rather than verify how program evaluation impacts on the 11 program components and stakeholders. For this purpose, qualitative evaluation was used to address the second and third research questions through the collection, analysis and interpretation of spoken and written discourse about program evaluation impact in order to use the resulting information for program improvement (Shawer 2012). Both quantitative and qualitative evaluation research assess program effectiveness, including planning, implementation, instructional methods, curriculum materials, facilities, equipment, educators and students better than other research strategies (Clarke 1999; Patton 1990; Shaw and Lishman 1999). Figure 1 further shows 10 questionnaires collected quantitative data from 16 faculty members who constituted the program population whereas seven administrative staff completed a questionnaire about student services. Quantitative data about faculty qualifications, teaching experience and professional development was also collected through program record and document analysis. In addition, student grade and completion records were analyzed to assess learning outcomes. Figure 1 also shows five questionnaires gathered data from randomly-selected 52 students. All questionnaires were developed, checked for reliability and content validated by the ministry of higher education (Cohen et al. 2011). Questionnaire reliability was further checked on 27 subjects. Using SPSS (version 14), alpha coefficients ranged between 81 and 89 (Coakes and Steed 2007). Figure 1 also shows semi-structured interviews (see the appendix) collected data from program director and five faculty members to assess the impact of program evaluation on

123

2888

S. F. Shawer

program improvement and stakeholders (Patton 1990). The interview data was content validated through some experts who ensured the questions addressed the research purpose (Bloom et al. 2009). Data reliability was checked through piloting and accuracy of transcribed texts. Interviews analysis involved developing concepts, categorization of similar concepts and forming a narrative (Kvale 1996).

6 Quantitative data analysis This section addressed the first research question through collecting evidence on each standard. 6.1 Standard 1: Mission, goals and objectives: (overall rating ***) We first brief the reader on how the data in Table 1 and other similar tables were analyzed. Table 1 (row 1) shows 16 faculty members’ responses to six requirements of the first substandard (appropriateness of the mission). We first summed each requirement responses, calculated each requirement percentages and summed the 16 percentages as 62.25 (row 1, column 2). We then calculated the average of percentages as 3.89 (62.25 ÷ 16) (row 1, column 3). Finally, the average was converted to 4 stars (row 1, final column), using a five-star scale (five stars = 4.5 and above, four stars = 3.6–4.5, three stars = 2.6–3.5, two stars = 1.6–2.5 and one star = 1.5 or below). It is clear 3.89 fell under 4 stars. This was repeated with the remaining rows. To get the overall assessment of the main standard (Mission, Goals, and Objectives), the average of sub-standards percentages was calculated as 3.2 (final row, column 3) and converted to 3 stars (final row, final column). Although Table 1 shows standard 1 was rated three stars, appropriateness of the mission received four stars where mission statement was appropriate for the program and consistent with the institute and university purposes. On the other hand, usefulness of mission statement and development and review of the mission received three stars (Table 1). Mission statement usefully guided program planning and decision-making, was achievable through available resources and provided criteria for evaluating progress against institutional goals. However, mission statement was not developed through consultative processes or periodically reviewed. As for use made of the mission, this sub-standard received just two stars (Table 1). The mission was not consistently used as a basis for planning, major policy decisions or strategic planning over a medium-term period (e.g. 5 years). Similarly, the mission was hardly publicized within the institute while little action was taken to ensure teaching faculty, staff and students knew about and supported it. Concerning relationship between mission and objectives, it was rated

Table 1 Faculty assessments of program mission and objectives No.

Item

% Total

% Average

Stars 4

1

Appropriateness of the mission

62.2

3.8

2

Usefulness of mission statement

56.7

3.5

3

3

Development and review of the mission

50.2

3.1

3

4

Use made of the mission

40.8

2.5

2

5

Relationship between mission & Objectives

48

3

3

3.2

3

Total

123

258

Accreditation and standards-driven program evaluation

2889

Table 2 Faculty assessments of program governance and administration quality No.

Item

% Total

% Average

Stars

1

Leadership

73.4

4.5

4

2

Planning processes

56.8

3.5

3

3

Integrity

71.2

4.4

4

4

Policies and regulations

Total

75.2

4.7

5

276.6

4.3

4

three stars (Table 1). The mission was the basis for defining goals and objectives. Despite concurring with the mission, medium and long-term goals were not developed. 6.2 Standard 2: Governance and administration: (overall rating ****) Table 2 shows standard 2 received four stars overall. Program administration generally provided effective and responsible leadership for development (four stars), where administrators had clearly-defined job responsibilities, anticipated issues and opportunities, exercised initiative and took actions in their area of responsibility in effective and timely manners. Academic affairs were monitored while allowing flexibility at course levels in terms of, for example, teaching strategy use. Table 2 shows planning processes received three stars. Planning processes were directed towards achieving program mission and goals through strategic planning with enough flexibility to adapt to changing circumstances. However, priorities were set for achieving effective short rather than long-term results. Despite monitoring plan implementation, checks were not made against short and medium-term outcomes. Senior administration rarely directed relevant personnel to review or adapt plans to take corrective actions in response to formative evaluation and changing circumstances. Table 2, however, shows high quality program integrity (four stars). The program met high ethical standards of honesty and integrity, including avoiding conflict of interest and plagiarism in teaching, research and service functions. Action was taken to ensure faculty, staff and students meet program standards and codes of practice with regulations applied to faculty, students, staff, governing board and committees. Hiring, disciplinary and dismissal practices were documented and administered fairly to all personnel. The program, however, did not review policies and procedures to ensure continuing high standards of ethical conduct. On the other hand, program policies and regulations received five stars. The program had comprehensive and widely accessible policies and regulations that establish terms of reference and procedures for committees, administrative units, positions and student responsibilities (codes of conduct). However, program policies and regulations were not periodically reviewed. 6.3 Standard 3: Management of quality assurance and improvement: (overall rating ***) Although Table 3 shows the third standard received three stars overall, there were differences between sub-standards ratings. Institutional commitment to quality improvement received four stars, where the dean strongly supported quality assurance improvement activities and provided adequate resources. Moreover, all teaching faculty and other staff conducted self-evaluations, cooperated with reporting and improvement processes and engaged in accountability processes periodically. Although mistakes and weaknesses were recognized, improvements in performance and outstanding achievements were hardly recog-

123

2890

S. F. Shawer

Table 3 Faculty assessments of program management of quality assurance and improvement No.

Item

% Total

% Average

Stars

1

Institutional commitment to quality improvement

58.4

3.6

4

2

Scope of quality assurance processes

51.2

3.2

3

3

Administration of quality assurance processes

67.3

4.2

4

4

Use of performance indicators and benchmarks

31.2

1.9

2

5

Independent verification of standards

40.0

2.5

2

248.1

3.1

3

Total

nized. Scope of quality assurance processes received three stars (Table 3), where academic and administrative units participated in quality assurance processes. Moreover, processes and outcomes were carried out and reports on performance were written. For example, students evaluated the program, courses, teaching faculty and recourses. In addition, performance of all program elements was evaluated against strategic objectives. Table 3 also shows administration of quality assurance processes received four stars, where a quality center was formally established and sufficient staff, resources and administrative support given to operate effectively. A quality committee was formed to run the center with roles and responsibilities clearly specified. The committee based all evaluations on evidence, linked them to appropriate standards and compared them against predetermined performance indicators. Common forms and instruments were standardized to gather evidence about various program elements and responses were independently analyzed. Statistical data including pass, progression and completion rates were retained in a data base. In contrast, the quality assurance arrangements were not regularly evaluated, reported on or improved. Nor were there independent verifications of procedures and interpretations. Table 3, however, shows a poor rating of use of performance indicators and benchmarks (two stars) where key performance indicators for undertaking objective measurement were rarely used to monitor and evaluate performance. Although benchmarks were established for comparing quality of performance of academic and administrative units, no comparisons were made between past and current performance. Nor were there external comparisons for important elements. Table 3, also shows another poor rating of independent verification of evaluations (two stars). Although self-evaluations of performance involved several instruments (e.g., questionnaires, interviews, record analysis) and stakeholder opinions (e.g., students, faculty and staff), conclusions were not independently verified. 6.4 Standard 4: Learning and teaching: (overall rating ***) Because standard 4 consisted of nine sub-standards, data was collected and analyzed from various sources to answer this question. The teaching faculty’s overall rating of these nine sub-standards was three stars (Table 4/final row and column). Ratings, however, differed across the nine sub-standards. Faculty members rated student learning outcomes (first sub-standard) three stars, where learning outcomes were defined through relevant academic and professional advice and covered all learning domains the standards required. Appropriate teaching strategies, student assessments and program evaluation mechanisms, including data collection and analysis from current and graduate students, teachers and employers, were also used. In contrast, the students rated student learning outcomes higher than their teachers. The students rated

123

Accreditation and standards-driven program evaluation

2891

Table 4 Faculty assessments of program learning and teaching quality No.

Item

% Total

% Average

Stars

1

Student learning outcomes

42.8

2.6

3

2

Program development processes

62.4

3.9

4

3

Program evaluation and review processes

55.9

3.4

3

4

Student assessment

54.8

3.4

3

5

Educational assistance for students

47.3

2.9

3

6

Quality of teaching

57.5

3.8

4

7

Support for improvements in quality of teaching

53.4

3.3

3

8

Qualifications and experience of teaching faculty

61.5

3.8

4

9

Partnership arrangements with other institutions

2.2

0.1

0

2.7

3

Total

438

Table 5 Student assessments of program quality No. Item

% Average % Average Sum of % Score of 5 Stars (strongly (agree) averages agree)

1

Quality of learning

65

24

89

4.45

2

Teaching and learning

65

31

96

4.8

4 5

3

Teaching support

64

26

90

4.5

4

4

Consultation & teaching support

34

33

67

3.35

3

5

Learning equipment, facilities & resources 61

29

90

4.5

4

6

Support for users of resources & IT

57

31

88

4.4

4

7

Appropriateness of course plans

45

39

84

4.2

4

learning quality four stars (Table 5/row 1) while rating teaching and learning processes five stars (Table 5/row 2). Moreover, Table 6 shows that students rated student learning outcomes almost five stars. We also brief the reader on how the data in Table 6 and similar tables was analyzed. Table 6 shows level-two, level-three and level-four students’ assessments of all courses. For each course, we grouped the 24 questionnaire items into just three categories (column 1). Concerning the Lang 120 course, student responses to first category (good plan and advising) were summed, a percentage of 86 (row 1, column 3) calculated and converted into 4.3 ((86 × 5 ÷ 100)/(row 1, column 4)) on a five-point scale between 1 and 5. The converted score was further converted to stars, where 4.3 fell between 3.6 and 4.5, indicating 4 stars (row 1, column 5). Finally, the overall assessment of the Lang 120 course was made by calculating the three categories average as 4 stars (row 1, column 6). The same process was repeated with the remaining categories and levels (Shawer 2012). The student academic records also support the high quality of student learning where Table 7 shows student program and level completion rates were between four and five stars. In addition, Table 8 shows 80% (4 stars) of student grades across level-one courses and 72% (also 4 stars) of grades across level-two courses fell between grades B and A+. Table 9 also shows that 66% (3 stars) of grades across level-three courses and 72% (4 stars) of grades across level-four courses were between grades B and A+.

123

123

Good plan & advising Good implementation Good learning Good plan & advising Good implementation Good learning Good plan & advising Good implementation Good learning Good plan & advising Good implementation Good learning Good plan & advising Good implementation Good learning Good plan & advising Good implementation Good learning Good plan & advising Good implementation Good learning Good plan & advising Good implementation Good learning Good plan & advising Good implementation Good learning

Category

Lang 126

Lang 125

Lang 124

Lang 123

Lang 122

Lang 127

Lang 120

86 79 80 96 94 94 96 97 97 96 89 97 98 97 97 98 97 98 100 100 93

4.3 3.95 4 4.8 4.7 4.7 4.8 4.9 4.9 4.8 4.4 4.9 4.9 4.9 4.9 4.9 4.9 4.9 5 5 4.7

4 4 4 5 5 5 5 5 5 5 4 5 5 5 5 5 5 5 5 5 5

Stars

Total stars

5

5

5

5

5

5

4

Lang 139

Lang 138

Lang 137

Lang 136

Lang 135

Lang 134

Lang 133

Lang 132

Lang 130

Course

Score of 5

Course

%

Level 3

Level 2

Table 6 Student assessments of courses at levels 2, 3 and 4

98 96 93 89 86 85 100 97 92 98 97 93 98 99 94 87 92 90 96 98 96 90 83 90 100 99 99

% 4.9 4.8 4.7 4.4 4.3 4.25 5 4.9 4.6 4.9 4.9 4.7 4.9 4.95 4.7 4.35 4.6 4.5 4.8 4.9 4.8 4.5 4.15 4.5 5 4.95 4.95

Score of 5 5 5 5 4 4 4 5 5 5 5 5 5 5 5 5 4 5 4 5 5 5 4 4 4 5 5 5

Stars

5

4

5

4

5

5

5

4

5

Total stars

Lang 149

Lang 148

Lang 147

Lang 146

Lang 145

Lang 144

Lang 143

Lang 140

Course

Level 4

99 95 100 99 95 100 98 94 100 98 94 98 99 98 50 90 92 86 99 95 100 99 95 98

% 4.95 4.75 5 4.95 4.75 5 4.9 4.75 5 4.9 4.7 4.9 4.95 4.9 2.25 4.5 4.6 4.3 4.95 4.75 5 4.95 4.75 4.9

Score of 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 2 4 5 4 5 5 5 5 5 5

Stars

5

5

4

4

5

5

5

5

Total stars

2892 S. F. Shawer

Accreditation and standards-driven program evaluation

2893

Table 7 Program and level completion rate Program Year

Students commenced the program in this group 2 years ago (2008–2009)

2009–2010 Level

Year

Students completed the program

17 Students commenced this level

Completion rate Score of 5 Stars

14

82

Students completed this level

4.1

4

Completion rate Score of 5 Stars

Level 1

2008–2009

17

15

88

4.4

Level 2

2008–2009

15

14

93

4.6

4 5

Level 3

2009–2010

14

14

100

5

5

Level 4

2009–2010

14

14

100

5

5

Table 8 Students’ grades at levels 1 and 2 Courses

Level

No.

A+

A

B+

B

C+

C

D+

D

F

Lang 112

Level 1

17 students

3

8

3

1

0

0

0

0

2

Lang 113

9

3

1

1

0

0

0

2

1

Lang 114

4

5

3

0

0

1

0

0

4

Lang 115

4

3

3

3

0

0

1

0

3

Lang 116

7

3

1

1

2

0

0

0

3

Lang 117

7

8

0

1

0

0

0

0

1

Lang 118

5

1

4

3

2

1

0

0

1

Lang 119

12

2

0

0

2

0

0

0

1

Total

51

33

15

10

6

2

1

2

16

Percentage

38

24

11

7

4

1

1

1

13

Grouped percentage

80

20

Score of 5

4

1

Stars

4

1

Lang 120

Level 2

15 students

5

3

2

1

3

0

0

0

1

Lang 127

1

6

4

0

2

0

0

1

1

Lang 122

6

3

2

0

0

1

1

0

2

Lang 123

6

2

3

0

0

2

0

0

2

Lang 124

4

1

2

2

3

1

1

0

1

Lang 125

5

6

0

0

0

2

0

0

2

Lang 126

4

3

1

4

0

0

1

2

0

Total

31

24

14

7

8

6

3

3

9

Percentage

29

23

13

7

8

6

3

3

8

Grouped percentage

72

28

Score of 5

3.6

1.4

Stars

4

1

123

2894

S. F. Shawer

Table 9 Students’ grades at levels 3 and 4 Courses

Level

No.

A+

A

B+

B

C+

C

D+

D

F

Lang 130

Level 3

14 students

5

3

0

2

3

1

0

0

0

Lang 132

3

4

2

1

1

2

1

0

0

Lang 133

1

4

3

1

3

0

0

1

1

Lang 134

10

2

1

0

0

0

0

1

0

Lang 135

2

4

3

0

1

1

0

2

1

Lang 136

1

1

2

0

0

2

2

2

4

Lang 137

9

3

0

0

1

0

1

0

0

Lang 138

1

3

2

0

1

2

2

0

3

Lang 139

0

1

3

6

2

1

0

0

1

Total

32

25

16

10

12

9

6

6

10

Percentage

25

20

13

8

9

7

5

5

8

Grouped percentage

66

34

Score of 5

3.3

1.7

Stars

3

Lang 140

Level 4

14 students

2

2

1

3

0

0

4

1

1

2

Lang 143

3

7

0

1

0

1

1

1

0

Lang 144

4

1

1

2

3

0

0

0

3

Lang 145

5

3

2

1

0

0

0

0

3

Lang 146

3

2

1

3

1

0

1

2

1

Lang 147

2

7

1

1

1

0

0

1

1

Lang 148

7

2

2

0

1

2

0

0

0

Lang 149

2

6

3

2

0

0

0

0

1

Total

28

29

13

10

6

7

3

5

11

Percentage

25

26

11

10

5

6

3

4

10

Grouped percentage

72

28

Score of 5

3.6

1.4

Stars

4

1

As shown in Table 4 above, faculty members rated program development processes four stars. Program delivery and evaluation plans were detailed through program and course specifications that set out the knowledge and skills to be acquired and teaching and assessment strategies. Planning included measures to ensure instructors were familiar with and able to use suggested teaching and assessment strategies. All courses were regularly monitored through ongoing evaluations and course reports. This resulted in developing new teaching series, a highly equipped language lab and the quality assurance center in addition to changing several textbooks. However, leading practitioners from relevant professions had not been invited to monitor and advise on program content and quality. Faculty members rated program evaluation and review processes three stars (Table 4 above), where program and all courses were monitored regularly through appropriate mechanisms. This included student assessments of program quality, program plan, courses and course instructors alongside graduate surveys and faculty assessments of program elements (e.g., teaching and learning). Courses and the program as a whole were evaluated and reported on each semester and annually. Based on gathered evidence, areas needing improvements

123

Accreditation and standards-driven program evaluation

2895

were amended while changes made were archived in course portfolios. Records of student completion rates were kept for the program and courses to indicate quality and program report reviewed annually. Despite ongoing annual and quarterly review, a comprehensive program reassessment was not conducted for over 8 years while all program reviews were internal. Faculty members rated student assessment three stars (Table 4 above), where student assessment processes suited intended learning outcomes and fairly administered but standards achieved were not independently verified. Student assessment mechanisms, including initial, formative and summative assessments through tests and portfolios were used to measure target learning outcomes. Likewise, assessment practices were clearly communicated to students at the beginning of courses in the course specification whereas the grading system was consistent over time and comparable across all courses. However, verifying standards through check marking of student work by teaching faculty from other institutions and independent comparisons of standards with other comparable institutions was not met. Feedback to students on their performance was given promptly but unaccompanied by mechanisms for assistance. Criteria and processes for academic appeals were made known to students and administered equitably. On the other hand, Table 10 (good learning assessment category) shows students rated assessment of their learning across all courses between four and five stars. Faculty members also rated educational assistance for students three stars (Table 4 above), where effective systems to assist student learning through academic advice, study facilities, monitoring progress, encouraging high performing students and provision of assistance when needed were employed. Each faculty member was available at sufficient scheduled times for 10 office hours a week. Teaching resources, including faculty, learning resources and equipment were also sufficient. Moreover, students received adequate tutorial assistance alongside providing orientation to new students and administering placement tests to ensure language skills are adequate for instruction. Although each student was assigned an academic advisor, the monitoring system did not really monitor or coordinate student workload across courses. In contrast, Table 5 above (rows 3–6) shows students rated educational assistance for students four stars. As shown in Table 4 above, faculty members rated quality of teaching four stars, where effective orientation and training were provided for new faculty through various development activities. Faculty members were briefed on required learning outcomes and planned teaching and assessment strategies. Teaching and assessment strategies set out in program and course specifications were followed by teaching faculty with flexibility to meet different needs. Any changes from predetermined plans were reported on and justified in each course report. Textbooks and reference material were updated and available in sufficient quantities before classes commenced, attendance requirements were made clear to students and compliance was strictly monitored. Moreover, students regularly evaluated courses and teaching with reports provided to program administrators on each course. However, appropriate adjustments in plans for teaching were hardly made after consideration of course reports. In contrast, Table 6 above shows students rated quality of teaching between four and five stars across all courses. The students provided very positive assessments of course plans (specifications), faculty advising, and implementation of planned topics, teaching strategies and assessment of learning. In particular, Table 10 shows students also rated course instructors between four and five stars. They agreed their instructors taught lessons and assessed learning outcomes effectively. In addition, instructors were punctual, had good personalities and encouraged them to learn. Faculty members rated support for improvements in quality of teaching three stars (Table 4 above). Ongoing training in teaching skills for new and continuing faculty was provided every

123

123

Good lesson teaching Good learning assessment Punctuality & Good personality Good lesson teaching Good learning assessment Punctuality & Good personality Good lesson teaching Good learning assessment Punctuality & Good personality Good lesson teaching Good learning assessment Punctuality & Good personality Good lesson teaching Good learning assessment Punctuality & Good personality Good lesson teaching Good learning assessment Punctuality & Good personality Good lesson teaching Good learning assessment Punctuality & Good personality Good lesson teaching Good learning assessment Punctuality & Good personality Good lesson teaching Good learning assessment Punctuality & Good personality

Category

Lang 126

Lang 125

Lang 124

Lang 123

Lang 122

Lang 127

Lang 120

81 84 81 100 100 100 93 100 92 100 100 100 92 100 88 100 100 100 97 100 93

4.05 4.2 4.05 5 5 5 4.65 5 4.6 5 5 5 4.6 5 4.4 5 5 5 4.85 5 4.65

4 4 4 5 5 5 5 5 5 5 5 5 5 5 4 5 5 5 5 5 5 5

5

5

5

5

5

4

Lang 139

Lang 138

Lang 137

Lang 136

Lang 135

Lang 134

Lang 133

Lang 132

Lang 130

Score of 5 Stars Total stars Course

Course

%

Level 3

Level 2

Table 10 Student assessments of course instructors at levels 2, 3 and 4

99 92 100 90 93 81 100 100 100 99 97 97 100 100 100 91 94 83 99 93 100 92 94 89 91 93 88

% 4.95 4.6 5 4.5 4.65 4.05 5 5 5 4.95 4.85 4.85 5 5 5 4.55 4.7 4.15 4.95 4.65 5 4.6 4.7 4.45 4.55 4.65 4.4

5 5 5 4 5 4 5 5 5 5 5 5 5 5 5 5 5 4 5 5 5 5 5 4 5 5 4 5

5

5

5

5

5

5

4

5

Lang 149

Lang 148

Lang 147

Lang 146

Lang 145

Lang 144

Lang 143

Lang 140

Score of 5 Stars Total stars Course

Level 4

98 100 98 98 100 98 98 100 98 97 100 97 100 100 100 98 100 98 98 100 97 98 97 98

% 4.9 5 4.9 4.9 5 4.9 4.9 5 4.9 4.85 5 4.85 5 5 5 4.9 5 4.9 4.9 5 4.85 4.9 4.85 4.9

5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5

5

5

5

5

5

5

5

5

Score of 5 Stars Total stars

2896 S. F. Shawer

Accreditation and standards-driven program evaluation

2897

Table 11 Teacher development activities, qualifications and experience Teacher

Training activities Number

1

2

2

3

3

3

4

3

5 6

Doctorate Master Diploma Experience Research

Training topic

0

English language Multimedia in teaching Teaching skills Academic leadership –

Quality assurance Academic advising Quality assurance Higher education –

0

–

7

0

8 9



0

0

28

3



0

28

0



0

0

15

1

Blackboard 0 Strategic planning Statistical analysis –



0

0

15

1

0



0

29

0

–

–

0



0

27

0

–

–

–



0

0

29

0

0

Blackboard

–

–

0



0

6

0

1

–

–

–



0

0

32

0

10

0

–

–

–



0

0

4

0

11

0

–

–

–

0



0

24

0

12

2

Blackboard

0



0

3

0

13

0

–

Quality assurance –

0

0



22

0

14

2

Quality assurance

English lan- – guage



0

0

28

0

7

6

1

290

5

Total

16

–

semester by the Deanship of Skills Development. In the same vein, the Deanship of Electronic Transactions provided ongoing training in effective technology use in education. However, all training activities were voluntary and faculty development was never monitored. Table 11 shows faculty members voluntarily completed 16 training sessions that ranged between teaching skills and integrating technology into teaching. Table 4 further shows faculty members rated qualifications and experience of teaching faculty four stars. Table 11 shows 50% of instructors obtained relevant doctorates, 43% Masters and 7% diplomas. They were also experienced since average experience was 21 years. However, Table 11 shows almost all teaching faculty hardly conducted scholarly activities to remain up-to-date with the latest developments in their fields. Their research output was sparse since only three members (21%) published five papers while 79% did not publish a single research paper. Table 4 finally shows faculty members rated partnership arrangements with other institutions no stars. No part of the program was delivered through cooperative arrangements with other programs. Nor were steps taken to look at the courses and requirements of other programs to establish useful partnerships. 6.5 Standard 5: Student administration and support services: (overall rating ****) Table 12 shows the administrative staff rated student administration and support services four stars overall. They rated student admissions three stars, where student admission requirements were clearly specified, appropriate and consistently and fairly applied. Program and course registrations were simple to use through a computerized system linked to data recording and

123

2898

S. F. Shawer

Table 12 Staff assessments of student administration and support services in the program No.

1 2 3 4

Item

Student admissions Student records Student management Medical & counseling services

Student affairs personnel

% Total

% Average

Stars

1

2

3

4

5

6

7

3.28

3.28

3.75

4

3.28

4

3.28

24.8

3.5

3

3.5

4

3.75

3.6

3.75

3.28

3.75

25.6

3.6

4

3.28

4.16

4.33

4.16

4.33

3.28

4.33

27.8

3.9

4

4

4.25

3.28

4.25

4

3.28

4.25

27.3

3.9

4

3.7

4

Total

132

retrieval systems. Students were issued identity cards straight away while course advisors provided assistance prior to and during registration. Complete information about the institute and program, including range of courses, program requirements, costs, services and other relevant information were publicly available to potential students prior to admission. Moreover, orientation programs were available to introduce new students to program services, facilities and their obligations and responsibilities. The administrative staff rated student records four stars (Table 12), where student records were secured, confidential and automated for generation of statistical data needed to provide performance indicators and generate progress and achievement reports. Eligibility for graduation was formally verified with results finalized, officially approved and communicated to students according to predetermined times. However, formal policies specifying content of permanent records, their retention and disposal were unavailable. The administrative staff also rated student management four stars (Table 12). Policies and regulations set out fair and consistent processes of student management alongside effective safeguards for independent consideration of disputes and appeals. A code of behavior specifying student rights and responsibilities was available to all students in addition to regulations that specify action to be taken for breaches of discipline. Moreover, impartial appeal and grievance procedures were made widely known. In addition, procedures protected students against subsequent punitive actions or discrimination following grievances or appeals. The policies also spelled out procedures in dealing with academic misconduct, including plagiarism and other forms of cheating. The administrative staff further rated medical and counseling services four stars (Table 12), where fully-qualified individuals provided accessible, free and confidential medical and counseling services for students in need. Provision was made for academic counseling, career planning and employment advice. Similarly, easy access to personal and psychological counseling services was available alongside effective mechanisms for follow up to ensure student welfare and to evaluate service quality. 6.6 Standard 6: Learning resources: (overall rating ****) Table 13 shows program learning resources received four stars overall. Faculty members rated planning and evaluation three stars (Table 13), where policies and procedures were in place to ensure materials and services were adequate and appropriate to support student learning. Adequacy of library and resource center materials were monitored and evaluated regularly

123

Accreditation and standards-driven program evaluation

2899

Table 13 Faculty assessments of learning resources in the program No.

Item

1

Planning and evaluation

% Total

% Average

Stars

56.7

3.5

3

2

Organization

69.8

4.3

4

3

Support for users

73.1

4.5

4

4

Resources and facilities

Total

78.2 278

4.8

5

4.3

4

against user needs, consistency with requirements of teaching and learning and range of services. However, they did not involve comparisons with other comparable programs or institutions. Moreover, organization received four stars (Table 13), where secured libraries and resource centers were available for extended hours. Collections were also arranged and catalogued according to internationally recognized good library practice alongside cooperation with other libraries and resource centers for sharing resources and services. Recording loans and returns was managed through an efficient system that follows up overdue material. There was also reliable and efficient access to on-line data-bases and research and journal material relevant to the program. Rules for behavior within the library were established and strictly enforced with effective security systems used to prevent loss and inappropriate use of resources. As shown in Table 5 above, the students also rated resources organization four stars. Faculty members also rated support for users four stars (Table 13). Library and learning resources staff provided adequate support to assist students and faculty to make effective use of library services and resources while Deanship of Electronic Transactions provided regular orientation and training programs for new users to prepare them to access facilities and services. Moreover, electronic and automated systems alongside sufficient qualified and skilled library staff in librarianship and information technology were available to help users search for, locate and use information. As also shown in Table 5 above, the students again rated support for users four stars, which concurs with faculty assessments. Finally, resources and facilities received five stars (Table 13). Resources and facilities were adequate for program learning and research requirements together with adequate financial resources for acquisitions, cataloguing, equipment, services and system development. Likewise, updated computer equipment and software were available to support electronic access to resources and reference material in addition to copying and personal laptop facilities. Books, journals and other materials were available. There were subscriptions to almost all relevant databases on electronic books and journal resources. However, provision of facilities and resources was not benchmarked. Again, the students rated resources and facilities four stars (Table 5 above), which agreed with faculty assessments. 6.7 Standard 7: Facilities and equipment: (overall rating ***) Table 14 shows program facilities and equipment received three stars overall. Faculty members rated policy and planning three stars (Table 14) where the university implemented master plans for managing facilities and equipment to meet the needs of all colleges and programs. Moreover, the university board approved a long-term master plan to provide for capital developments and maintenance of facilities and equipment. In addition, business plans with evaluation of alternatives were made prior to major equipment acquisitions. Although planning addressed stakeholders’ requirements, stakeholders were not consulted in plans development.

123

2900

S. F. Shawer

Table 14 Faculty assessments of facilities and equipment in the program No.

Item

% Total

% Average

Stars

1

Policy and planning

42.2

2.6

3

2

Quality and adequacy of facilities

44.9

2.8

3

3

Management and administration

67.1

4.1

4

4

Information technology

65.6

4.1

4

219.8

3.4

3

Total

Table 15 Faculty assessments of financial planning and management in the program No.

Item

% Total

% Average

Stars

1

Financial planning and budgeting

70.6

4.4

4

2

Financial management

60.9

3.8

4

131.5

4.1

4

Total

Faculty members also rated quality and adequacy of facilities and equipment three stars (Table 14). A clean, attractive and well-maintained physical environment of buildings and grounds was maintained and facilities met health and safety requirements. Appropriate religious observances and food service facilities were available and the needs of physical disability personnel adequately addressed. Likewise, adequate facilities were available for cultural, sporting and extracurricular activities. The students, however, rated quality and adequacy of facilities and equipment four stars (Table 5 above). The teaching faculty, however, allocated management and administration of facilities and equipment four stars (Table 14). Services, such as cleaning, waste disposal, minor maintenance, safety, and environmental management were maintained efficiently and effectively. Effective security for facilities and equipment was provided with responsibility between individual faculty, departments and colleges and central administration clearly defined. Although equipment was replaced when required, regular condition assessments were neither carried out nor provision for preventative and corrective maintenance. Like faculty members, the students also rated management and administration of facilities and equipment four stars (Table 5 above). Faculty members further assigned information technology four stars (Table 14). Computing equipment, software and related support services were adequate and managed by Deanship of Electronic Transactions in ways that ensured secure, efficient and effective utilization. Adequate computer equipment was also available and accessible for teaching, staff and students but provision of computer equipment was not benchmarked. Again, like faculty members, the students rated information technology four stars (Table 5 above). 6.8 Standard 8: Financial planning and management: (overall rating ****) As summarized in Table 15, faculty members rated financial planning and management four stars overall. They rated financial planning and budgeting four stars (Table 15), where budgeting and resource allocations were aligned with the mission and goals and strategic planning. Senior academic and administrative staff in consultation with cost center managers carefully developed and reviewed budget proposals for governing body approval. Moreover,

123

Accreditation and standards-driven program evaluation

2901

Table 16 Faculty assessments of employment processes in the program No.

Item

1

Recruitment

2

Personal and career development

Total

% Total

% Average

Stars

73.6

4.6

5

74

4.6

5

148.5

4.6

5

% Total

% Average

Stars

Table 17 Faculty assessments of research processes in the program No.

Item

1

Faculty involvement in research

40

2.5

2

2

Research facilities and equipment

62.4

3.9

4

102.4

3.2

3

Total

new program and major activity equipment proposals were accompanied by business plans that include independently verified cost estimates and cost impacts on other services and activities. Financial management was also allocated four stars (Table 15), where financial affairs were effectively managed with proper balance between local flexibility for cost center managers and institutional accountability and responsibility. 6.9 Standard 9: Employment processes: (overall rating *****) As shown in Table 16, faculty members rated employment processes five stars overall. Program records summarized in Table 11 above also confirm this rating. Faculty members rated recruitment five stars (Table 16), since recruitment processes ensured instructors and staff were capable and appropriately qualified and that new faculty and staff were thoroughly prepared for their responsibilities. Instructors had the specific areas of expertise, personal qualities, experience and skills to meet teaching requirements. All appointments made through promotion or transfer met qualification and skill requirements while appointment procedures ensured equitable treatment of all applicants through public advertising of positions. New faculty and staff were given orientation to ensure familiarity with services, programs, student development strategies and institutional priorities for development. Faculty members also rated personal and career development five stars (Table 16), where processes for personal and professional development were available, frequent and fair to all faculty and staff. All faculty and staff had appropriate and fair opportunities for personal and career development through the ongoing courses offered by Deanship of Skills Development and Deanship of Electronic Transactions. Criteria and processes for performance evaluation were specified and made known in advance. Moreover, consultations about work performance were confidential and supportive. Clear requirements for improving unsatisfactory performance were established while performance assessments were confidential, documented and retained. Faculty provided comments about assessments on their files, including points of disagreement. Although outstanding academic and administrative performance was recognized and rewarded at the university level, this was not the case at the program level.

123

2902

S. F. Shawer

Table 18 Faculty assessments of program relationships with the community No.

Item

1

Institutional policies on community relationships

2

Interactions with the community

Total

% Total

% Average

Stars

62.2

3.8

4

6.8

0.4

0

69.1

2.1

2

6.10 Standard 10: Research: (overall rating ***) Table 17 shows faculty members allocated research processes three stars overall. They rated faculty involvement in research two stars. As summarized in Table 11 above, program records also confirm this poor rating since only three faculty members published related papers. Table 11 further shows a very sparse research output where a single faculty member published three papers while the other two published a paper each. On the other hand, Table 17 shows faculty members agreed that expectations for teaching faculty involvement in research and scholarly activities were not specified in the program. Nor were there performance evaluation and promotion criteria in relation to scholarly activities of faculty to keep their jobs. In contrast, faculty members allocated research facilities and equipment four stars since the university provided adequate facilities and equipment appropriate for research in all fields of study. This included space, equipment, library and information systems and resources to support research activities. The university provided support for developing research programs through mechanisms, such as mentoring by senior colleagues, funded project teams and assistance in developing research proposals. However, the program did not take action to make use of university research facilities. 6.11 Standard 11: Relationships with the community: (overall rating **) As summarized in Table 18, faculty members rated this standard two stars overall. On one hand, faculty members rated institutional policies on community relationships four stars, since community service was a main part of both the university and institute missions and supported by senior administrators. On the other hand, faculty members rated interactions with the community zero stars (Table 18). Neither did the program draw on available expertise to contribute to community nor was community expertise drawn upon to strengthen the program. Teaching faculty and other staff were rarely encouraged to participate in forums in which significant community issues discussed and plans for development considered. Moreover, the program did not offer community education courses in areas of interest and need. In addition, local employers and members of professions were never invited to join appropriate activities and advisory committees. Even schools in the community were never contacted to assist and support in specialization areas. Likewise, no contact was made with alumni to keep them informed about program developments. Finally, Table 19 shows the overall rating of each of the 11 standards as well as their three-star combined rating.

7 Qualitative data analysis This section addressed the second and third research questions through collecting qualitative evidence about program evaluation impact on each program element and stakeholders.

123

Accreditation and standards-driven program evaluation

2903

Table 19 Overall assessments of the program against the 11 standards No.

Standard

Score average

Stars

1

Mission, goals and objectives

3.2

3 4

2

Governance and administration

4.3

3

Management of quality assurance and improvement

3.1

3

4

Learning and teaching

2.7

3

5

Student administration and support services

3.7

4

6

Learning resources

4.3

4

7

Facilities and equipment

3.4

3

8

Financial planning and management

4.1

4

9

Employment processes

4.6

5

10

Research

3.2

3

11

Program relationships with the community

2.1

2

Total: 38.7 Average: 3.5

3

Overall program quality on the 11 standards

7.1 Program evaluation impact on program elements This section addressed the second research question. 7.1.1 Program evaluation impact on program mission, goals and objectives The respondents agreed program evaluation contributed to improving program mission, goals and objectives. “For the first time we have understood what a mission means. The program did not actually have a mission in the real sense. Not only did program evaluation force the program to have a mission, but also the program had to make the mission consistent with the institute and university purposes.” They agreed that “in the past, we had a mission for the sake of a mission. It did not have any impact on what is going on. Now, the mission guides the whole program. We now understand that any decisions should help realize the mission.” The respondents further agreed that “we refined our goals and objectives in line with the mission because from now on our performance would be assessed against these goals and objectives.” One was concerned that “we don’t have a say in mission or objectives formulation but we are happy now that this program evaluation process stipulates that we should contribute to any future program review. To be honest, this broadened our horizons on many issues that we either did not know or we did it in unprofessional ways.” 7.1.2 Program evaluation impact on program governance and administration Faculty members and program director all agreed program evaluation has improved program governance and administration. “Although program management was fine, it has become more effective. We notice that program director is now more concerned with improving all program elements. Tasks are set, roles are defined and we all now held accountable against these defined tasks.” They agreed, “responsibilities are now very defined and a timeframe for achieving tasks is now chasing us up. In the beginning, I acknowledge this was very stressing and sometimes impossible but now we know what to do and how to do it. By emulating and developing what we have done, the process has become manageable.” Moreover,

123

2904

S. F. Shawer

“academic affairs including program and course specifications and reports are now under ongoing scrutiny.” In addition, “we collaborate in achieving program goals and objectives. This was once unthinkable. Everyone worked in isolation, but we are now individually and collectively accountable. We are very happy to see the institute’s strategic plan recognized among the best plans by the university. This is very encouraging.” All respondents shared this statement. “The program now has a strategic plan, set priorities, and plans are widely and efficiently communicated. We now look at what is not working and look into improving it.” 7.1.3 Program evaluation impact on program management of quality assurance and improvement There was consensus that “program administration has become committed to quality improvement. The students now assess all program elements, including program administration.” They agreed “in every course, the students assess the teaching faculty, course content, teaching strategies, academic advising, methods of assessing learning and the quality of learning they have achieved.” Moreover, “the students also assess all program resources, including library, computers, classrooms and electronic services and support.” In the same vein, the respondents noted “forming the program quality assurance committee was one of the major outcomes of program evaluation.” The influence of this committee was huge where “standardized instruments were used to gather data about program performance from all program stakeholders. The data was analyzed and fed back to program administration to take action. This has never been done before.” 7.1.4 Program evaluation impact on program learning and teaching Program evaluation also contributed to learning and teaching, since “program learning outcomes were defined in line with the National Qualifications Framework to include all learning domains required by the standards.” We had to “spell out expected teaching strategies and learning outcomes in program and course specifications.” All faculty members “had to demonstrate familiarity with and ability to use suggested teaching and assessment strategies.” Program evaluation has also engendered “development and use of new teaching series, offering in-service training on effective teaching, assessment of student learning and use of blended learning. We are told that the next semester, all faculty members have to offer 20% of their courses by distance learning through the Blackboard.” They shared this statement. “Program evaluation has forced everybody to spell out a detailed plan of what to teach and how to teach and assess it in the course specification. We are forced even to align the course and program objectives and outcomes.” They added “what we considered too much at the beginning of the process is that we had to justify the topics that were not actually taught and suggest alternative ways of covering them. We also had to include and respond to student assessments in our course reports. This posed real challenges to us all and put us under the spotlight.” As the respondents acknowledge improvement of teaching processes, they also agreed “student learning has improved as a result of improvements in program content, resources and teaching planning and implementation.” 7.1.5 Program evaluation impact on student administration and support services The respondents shared positive statements about program evaluation influence on student administration and support services. “The program became more clear, consistent, precise and

123

Accreditation and standards-driven program evaluation

2905

transparent about student admission requirements. The program used a computerized system for course registration and assigned course advisors to assist students during registration.” Moreover, alongside program manual, the program started to “provide orientation to introduce new students to program services, facilities and their obligations and responsibilities.” In addition, “better procedures were used to keep student records secure and confidential, eligibility for graduation was formally verified and results communicated to students as specified in the program manual. The students now appeal without fear of punishment as impartial committees now tackle their grievances and appeals.” 7.1.6 Program evaluation impact on program learning resources, facilities and equipment The respondents agreed that availability and quality of resources, facilities and equipment “are controlled by the university and institute,” and that “the university with its huge resources and budget provides the best resources in the world. All classrooms are equipped with smart and traditional boards. Special need facilities are available everywhere, including restrooms, parking and elevators.” Moreover, “the institute and university libraries and resource centers provide updated and sufficient hardware and software material, books and journals that address all program objectives.” All libraries and resource centers “were organized and available for extended hours beyond normal class time. There were sufficient study rooms, internet-enabled computers and support for users. The university has subscriptions to almost all major information sources that provide full text to books and journals.” The university and institute “provide sufficient and high quality facilities and equipment and provide ongoing maintenance and security.” Program evaluation, however, “did not have an impact on learning resources because they are sufficient, well-managed and of high quality.” 7.1.7 Program evaluation impact on program employment processes and professional development The respondents agreed that program evaluation did not make a difference concerning program employment processes since “the program employed systematic procedures in recruiting new staff and faculty. There was a committee that review applications and select a shortlist of candidates whose qualifications and experience match the job description.” After interviewing candidates, “the committee nominates the qualified people for approval by the program, the institute and finally the university.” On the other hand, program evaluation had a very positive influence on faculty and staff development. “Although training courses have been available on a weekly basis by the Deanship of Skills Development and Deanship of Electronic Transactions, we did not make use of them to develop our skills. Since program evaluation counts professional development as part of program assessment, we have to demonstrate we continue to develop our skills.” This led “program administration to count documented professional development as part of faculty assessment. This made difference.” 7.1.8 Program evaluation impact on program research processes and relationships with the community Program evaluation had a major impact on the program research processes. “We have been only concerned with teaching, but we did not care about undertaking research. Program evaluation again counted research processes and output as part of program assessment. Renewing

123

2906

S. F. Shawer

our contracts has become contingent partly on research publication.” By the same token, program evaluation “resulted in a big increase in funded collaborative research projects, which encouraged faculty members and instructors to conduct research.” Similarly, all respondents confirmed that program evaluation improved program relationships with the community. “As you know, the program did not relate to the community in any sense, but the program evaluation process counts the program relationships with the community. Now, contribution to community service has been part of faculty assessment. Everyone now has an agenda about how they can serve the community.” In addition, the program started to “put university regulations about community service into action. These days you see workshops and lectures by program and community members in many community centers and on campus.” 7.2 Program evaluation impact on program stakeholders This section addressed the third research question. The respondents made positive statements about the influence of program evaluation on their development of several professional skills. “This program review has made a huge difference in my career. It has answered several questions I have asked before but did not have the inclination or time to address them.” For them, program evaluation was like “ongoing professional development training, more or less a useful imperative to learn through teaching.” Although program review “was distressing and demanding, it made us more aware why we did things and for what purposes. We have been doing things for years but did not ask ourselves why we did them. Now we see the whole picture and understand what influences what and how to make things work.” The respondents shared similar concerns. “I have seen everyone unhappy with the sweep of procedures that overwhelmed us all. Out of the blue, we have been asked to address everything at one time. To be honest, we had to sacrifice teaching time to familiarize ourselves with new requirements that we had to meet.” Moreover, “we have been talking with one another about the ordeal that we all had. We did things just because we have been asked to do. We have never expected this program review would benefit us that much. Or would we support it. This work is investment.” The respondents gave examples of their learning as they went through program evaluation. “I have learned how to prepare the course specification, align course and program objectives and outcomes and assign course topics to semester weeks and hours. I have become more aware of what to assess, how to assess it and methods of assessing student learning.” Likewise “I now think of what content and teaching strategies are suitable to achieve course objectives. We had to specify the knowledge and skills that we seek to help our students to attain.” They agreed “the students made also much benefit for becoming clear about what to learn and how to be assessed. This helped them plan and manage their learning and effort.” One explained “we prepare our course reports and look at what went wrong with what we set out in our course specification and what procedures we needed to address weaknesses and challenges.” One added. “The course report brought to our attention that what we plan in our course specification would be checked out. We also learned how to analyze student assessments of our courses, interpret them and comment on student scores using descriptive statistics and simple graphs. This has been wonderful.” They further agreed they also “learned how to prepare a course portfolio through a course file report that includes teaching philosophy, student work, course specification, course report, evaluation of teaching, course content, and bank of questions.” One commented, “This helped us create a data base for all courses. I made use of this when I taught a new course, I made use of the course file elements my colleague has created.” Even the program director admitted that “although this program evaluation exhausted us and I personally was dismayed by

123

Accreditation and standards-driven program evaluation

2907

the news that we have to make this major review, I am happy with the results. Things have become clearer.” He added, “I used to make decisions without looking at the whole picture. Now, all my decisions are made to address program objectives and mission. Even this has not become sufficient since I have to address the compatibility between our mission and goals and those of the institute and university.” His overall impression was that “things have become completely different. I can say thank you program evaluation.”

8 Discussion We come back to answer and discuss our research questions. The study answered the first question to some extent in positive. The 11 standards received a combined overall rating of three stars. This means good program performance was maintained, the 11 standards were often addressed and evidence of the effectiveness about each standard’s activities was usually obtained. This also indicates that satisfactory standards of performance were normally achieved. Despite some room for improvement, improvement plans were made and progress was monitored. However, certain standards and sub-standards performance was unsatisfactory. Standard 11 received just two stars, which means major improvement was required because the quality was unsatisfactory. Other program components (standards 1, 3, 4, 7 and 10) received three stars, which indicates good performance. However, several sub-standards (use made of the mission (standard 1), use of indicators and benchmarks and independent verification of standards (standard 3), and faculty involvement in research (standard 10)) received just two stars and therefore performance was unsatisfactory. Of course, such poor ratings negatively influence main standards even if overall performance was satisfactory. Moreover, one sub-standard (partnership arrangements with other institutions (standard 4)) even received zero stars, which means performance was completely unsatisfactory. In contrast, other standards (standards 2, 5, 6 and 8) received four stars and therefore showed high quality performance. In addition, standard 9 was rated five stars and therefore showed very high quality performance. These findings concurred with most previous research conclusions where language programs showed overall satisfactory performance (e.g., Arnold 2009; Bernhardt 2006; Houston 2005; Rivera and Matsuzawa 2007; Topkaya and Kucuk 2010). However, these findings disagreed with a number of studies that found overall program performance less than satisfactory (e.g., Dassier and Powell 2001; Gorsuch 2009; Sanders 2005). Although the first research question was answered by looking at overall results of the 11 standards, we discuss the results of each standard below. Program mission, goals and objectives (first standard) received three stars. This means good performance because this practice was often followed and evidence of effectiveness was usually obtained. This also means satisfactory standards of performance were normally achieved. Although there was some room for improvement, plans for improvement in quality were made and progress in implementation was monitored. Despite the overall good performance of standard 1, the fourth sub-standard (use made of the mission) in particular was unsatisfactory (two stars). Although the practice was usually followed, the quality was less than satisfactory. This required major improvements in this particular sub-standard. Standard 1 findings to a large extent confirm those of Arnold (2009); Middlebrook (1991) and Topkaya and Kucuk (2010) who found program goals and objectives maintained satisfactory performance. In contrast, these results were in dissonance with some studies that showed less than satisfactory performance of goals and objectives (Dassier and Powell 2001; Rivera and

123

2908

S. F. Shawer

Matsuzawa 2007). In this respect, researchers may examine the influence of satisfactory and unsatisfactory program mission, goals and objectives on overall program performance. Program governance and administration (second standard) received four stars. This means high quality performance where this practice was followed consistently and indicators of quality of performance were established. Despite this overall high quality performance, there was still some room for improvement. For this purpose, improvement plans had been developed, were being implemented and progress was regularly monitored and reported on. Moreover, although sub-standards received high ratings between four and five stars, the second substandard (planning processes) was rated relatively low (three stars). These high quality performance findings agreed with previous research conclusions indicating overall satisfactory performance of program governance and administration (Chase 2006; Kiely and Rea-Dickins 2005; Middlebrook 1991). Program management of quality assurance and improvement (third standard) received three stars. Although this indicates overall good performance, there was disparity across the sub-standards. Both first (institutional commitment to quality improvement) and third substandards (administration of quality assurance processes) received four stars, which indicates high quality performance while scope of quality assurance processes (second sub-standard) received three stars, which indicates just good performance. In contrast, use of performance indicators and benchmarks and independent verification of evaluations received only two stars, which shows poor and less than satisfactory performance. Although this demands a lot of improvement in these particular two sub-standards, this negatively impacts on the main standard. Findings about this particular part of program evaluation (standard three) was not confirmed or disproved by previous research since a study assessing this particular element was not located. Future researchers need therefore to examine program management of quality assurance and improvement procedures impact on program performance. Program learning and teaching (fourth standard) was generally effective since the nine sub-standards that constitute this standard received a combined three-star rating. This means good performance with some room for improvement. Despite standard 4 overall good performance, there was big disparity not only between faculty ratings across the nine sub-standards but also between faculty and student ratings. Faculty members rated student learning outcomes (first sub-standard) three stars, but students rated it four and five stars. This means faculty ratings indicate good performance whereas student ratings show high and very high quality performance. Moreover, student grades at all courses also indicate high and very high quality performance since most grades fell between B and A+. Faculty members also rated student assessment (fourth sub-standard) three stars, while student ratings once more were four and five stars across all courses. Invariably, faculty members rated educational assistance for students (fifth sub-standard) three stars, but the students rated it four stars. Again, faculty ratings indicate just good performance whereas student ratings show high and very high quality performance. These findings about learning outcomes supported many studies that showed overall satisfactory program performance in terms of improved language proficiency (Arnold 2009; Bernhardt 2006; Houston 2005; Rivera and Matsuzawa 2007; Topkaya and Kucuk 2010), while disagreed with many other studies that showed less than satisfactory performance (Dassier and Powell 2001; Gorsuch 2009; Sanders 2005). Future researchers may therefore examine why some language programs help students improve language performance while others do not. In few cases like this one, faculty and student ratings indicate high and very high quality performance. Faculty members rated quality of teaching (sixth sub-standard) four stars, while students rated it four and five stars. On the other hand, faculty members rated program development processes (second sub-standard) and qualifications and experience of teaching

123

Accreditation and standards-driven program evaluation

2909

faculty (eighth sub-standard) four stars, which indicates high quality performance. Moreover, they rated both program evaluation and review processes (third sub-standard) and support for improvements in quality of teaching (seventh sub-standard) three stars, which suggests good performance. Although all faculty ratings across the nine sub-standards ranged between three and four stars, they allocated partnership arrangements with other institutions (ninth sub-standard) a very poor rating of zero stars. This indicates this practice, though relevant, was not followed at all. These findings were in line with those of Arnold (2009), Bernhardt (2006), and Rivera and Matsuzawa (2007) who showed overall satisfactory program quality of teaching while disagreed with Dassier and Powell (2001), Gorsuch (2009) and Sanders (2005) whose studies showed less than satisfactory program teaching quality. Researchers may also examine why some language programs maintain high teaching quality while others do not and assess the influence of this on language learning. Student administration and support services (fifth standard) received four stars overall. This means high quality performance. The sub-standards were also rated high where sub-standards two (student records), three (student management) and four (medical and counseling services) were assigned four stars. Even the first sub-standard (student admission) that was rated three stars was on the four-star borderline. These findings agreed with those studies indicating overall satisfactory performance of program administration and student support services, including admissions, orientation, advising, financial aid and housing (Dassier and Powell 2001; Middlebrook 1991; Topkaya and Kucuk 2010). Researchers may also investigate the relationships between student administration and support services and learning outcomes. Program learning resources (sixth standard) received four stars, indicating high quality performance. Like standard 5, standard 6 and its sub-standards received high ratings. Both faculty members and students rated organization of resources (second sub-standard) and support for users (third sub-standard) four stars. The teaching faculty also rated resources and facilities (fourth sub-standard) five stars, while their students assigned a close rating of four stars. Despite receiving three stars, the first sub-standard (planning and evaluation) was close to a four-star rating. This indicates high quality performance because not only the main and sub-standards ratings matched but also faculty and student ratings. These findings concurred with Arnold (2009), Elder (2009), Kiely and Rea-Dickins (2005), and Middlebrook (1991) who found overall satisfactory program resources in terms of library, resource centers, ratio of faculty to students, staff, equipment, buildings, classrooms and study rooms while contradicted those indicating less than satisfactory performance (e.g., Topkaya and Kucuk 2010). Program employment processes (ninth standard) received five stars. This means high quality performance where this practice was followed consistently, quality indicators were established and improvement plans were developed, being implemented and progress was regularly monitored and reported on. Like standards 5, 6 and 8, standard 9 and its sub-standards received typical ratings. Faculty members rated both recruitment (first sub-standard) and personal and career development (second sub-standard) five stars. These findings agreed with those studies that indicate satisfactory program employment processes (Middlebrook 1991) and professional development opportunities (Bernhardt 2006; Dassier and Powell 2001; Kiely 2006; Norris 2009). However, they disagreed with Topkaya and Kucuk (2010) who found program development opportunities less than satisfactory. Researchers may study the relationships between program professional development opportunities and quality of teaching and learning outcomes. Program research processes (tenth standard) received three stars. Although this rating indicates good performance, the discrepancy between sub-standards ratings cast doubts on

123

2910

S. F. Shawer

the overall rating. At the negative side, faculty members allocated faculty involvement in research (first sub-standard) a very poor rating of just two stars. Moreover, this poor rating was confirmed by program records that show faculty research output and activities almost nonexistent. This indicates faculty involvement in research was usually followed but the quality and quantity were less than satisfactory. At the positive level, research facilities and equipment (second sub-standard) received four stars, which means high quality performance. Although the program had excellent resources, why was program research unsatisfactory? Why did few faculty members engage in scholarly activities? The apparent explanation points to the fact that research contributions were not recognized or reflected in faculty evaluation and promotion criteria. Nor were research outputs monitored, reported on or benchmarked which gave the impression that program administration valued classroom instruction only. Program relationships with the community (eleventh standard) received two standards. At the positive side, faculty members rated institutional policies on community relationships (first sub-standard) four stars. Although this indicates high quality performance, unfortunately, this was at the university rather than program level. At the negative side, faculty members rated interactions with the community (second sub-standard) zero stars, which indicates this practice, though relevant, was not followed at all. Although the program and institution regulations set it a quality standard, no actions were taken in the program to put university regulations into action. It was true the program was interactive with the international community through granting scholarships to international students. However, the program was a complete failure at the national and local levels. Although the program was directed to international students, this does not mean the program continues in total isolation from the local community. These findings disagreed with Pawan and Thomalla (2006) who found satisfactory program relationships with the community. Having answered and discussed in detail the first research question, we now turn to answer and discuss the second and third research questions. Answering this second research question was very much in positive: What use did the program make of program evaluation? Apart from program governance and administration and employment processes, the program evaluation processes had resulted in significant improvements in all program elements. Even program elements that did not improve as a result of the program evaluation process this was due to having very high quality in the first place. These findings very much agreed with previous research results that found program evaluation brings about significant program improvements (Carsten-Wickham 2008; Dassier and Powell 2001; Kiely 2009; Norris 2009). Answering this third research question was also very much in positive: How did program evaluation impact on program stakeholders? Program director, faculty members and instructors all found program evaluation improved their professional skills. Program evaluation improved their ability to formulate course objectives and align them with program mission and goals. Program evaluation improved their skills of defining course skills and knowledge and helped them choose suitable teaching strategies and content that help realise their objectives. Their skills improved with regard to aligning course and program outcomes, assessing student learning and analyzing and interpreting results. These findings concurred with previous research that found program evaluation improves faculty professional skills (Carsten-Wickham 2008; Chase 2006; Kiely 2006; Norris 2009; Sullivan 2006). Although program faculty and director perceived program evaluation as a threat in the beginning, their perceptions turned positive as a result of their positive experiences. This concurred with Norris (2006, 2009) who concluded program evaluation is an opportunity for learning rather than a threat.

123

Accreditation and standards-driven program evaluation

2911

9 Conclusions, recommendations and limitations The study concluded that the language program achieved overall satisfactory standards of performance but some components showed better performance than others. Moreover, program evaluation brought about significant improvements in most program elements and improved faculty professional skills. The study recommends building evaluation of program evaluation as an integral part of any program evaluation. Future researchers need to examine in more systematic ways the influence of program evaluation on faculty and staff professional development as well as what factors lead program stakeholders to change their initial positive or negative attitudes towards imposed program evaluations. Ongoing program monitoring reminds administration, faculty, staff and students to direct effort and resources to achieve the standards. This improves program overall performance in general and teaching and learning in particular. This study’s findings should be, however, looked at with some caution since it was a self-evaluation study that represented one program. Although every effort was made to collect, analyze and interpret research evidence as objectively and rigorously as possible, lack of external assessments was a major weakness. Acknowledgements The author extends his appreciation to the Deanship of Scientific Research at King Saud University for funding this work as part of a large project through the research group grant number RGP-VPP-113.

Appendix: Interview main and follow up questions with program stakeholders 1.

How did the program evaluation process influence the program? • • • • • • • • • • •

2. 3. 4. 5. 6. 7.

How did it impact on program mission, goals and objectives? How did it impact on program governance and administration? How did it impact on program management of quality assurance and improvement? How did it impact on program teaching and learning processes? How did it impact on student administration and support services in the program? How did it impact on program learning resources? How did it impact on program facilities and equipment? How did it impact on program financial planning and management? How did it impact on program employment processes? How did it impact on program research processes and output? How did it impact on program relationships with the community?

What negative impact did the program evaluation process have on the program? What positive impact did the program evaluation process have on the program? How do you perceive the imposed program evaluation and accreditation processes? Did the program evaluation process have any influence on you? Do you think the program evaluation process was a threat or a learning opportunity? What did you learn from your experiences in the program evaluation process?

References Anderson, L., Olsen, B.: Investigating early career urban teachers’ perspectives on and experiences in professional development. J. Teach. Educ. 57(4), 359–377 (2006)

123

2912

S. F. Shawer

Arnold, N.: Online extensive reading for advanced foreign language learners: An evaluation study. Foreign Lang. Ann. 42(2), 340–366 (2009) Bernhardt, E.B.: Student learning outcomes as professional development and public relations. Mod. Lang. J. 90(4), 588–590 (2006) Bloom, M., Fischer, J., Orme, J.: Evaluation Practice: Guidelines for the Accountable Professional, 6th edn. Allyn and Bacon, Boston (2009) British National Standards Body: About Standards. http://corpslakes.usace.army.mil/employees/faqs.cfm? Id=perf-measure. Accessed 14 June 2011 Byrnes, H.: Perspectives. Mod. Lang. J. 90(iv), 574–576 (2006) Carsten-Wickham, B.: Assessment and foreign languages: a chair’s perspective. ADFL Bull. 39(2&3), 36– 43 (2008) Chase, G.: Focusing on learning: reframing our roles. Mod. Lang. J. 90(4), 583–588 (2006) Clarke, A.: Evaluation Research: An Introduction to Principles, Methods and Practice. Sage, London (1999) Coakes, S.J., Steed, L.: SPSS Version 14.0 for Windows: Analysis Without Anguish. Wiley, Milton (2007) Cohen, L., Manion, L., Morrison, K.: Research Methods in Education, 7th edn. Routledge, London (2011) Council for Higher Education Accreditation: The Fundamentals of Accreditation. CHEA, Washington (2002) Craft, A.: Continuing Professional Development: A Practical Guide for Teachers and Schools. Routledge, London (1996) Dassier, J.P., Powell, W.: Formative foreign language program evaluation: Dare to find out how good you really are. Dimension 2001: The odyssey continues. In: Selected Proceedings of the 2001 Conference of the Southern Conference on Language Teaching, Birmingham, AL, pp. 15–30, 2001 Elder, C.: Reconciling accountability and development needs in heritage language education: A communication challenge for the evaluation consultant. Lang. Teach. Res. 13(1), 15–33 (2009) European Network for Quality Assurance in Higher Education: Quality Assurance in the Nordic Higher Education—accreditation-like practices, ENQA Occasional Papers (2), Helsinki, Finland, p. 7. http://www. enqa.eu/files/nordicquality.pdf. Accessed 21 June 2011 Gorsuch, G.: Investigating second language learner self-efficacy and future expectancy of second language use for high-stakes program evaluation. Foreign Lang. Ann. 42(3), 505–540 (2009) Houston, T.: Outcomes assessment for beginning and intermediate Spanish: one program’s process and results. Foreign Lang. Ann. 38(3), 366–376 (2005) Kiely, R.: Evaluation, innovation, and ownership in language programs. Mod. Lang. J. 90(4), 597–601 (2006) Kiely, R.: Small answers to the big question: Learning from language programme evaluation. Lang. Teach. Res. 13(1), 99–116 (2009) Kiely, R., Rea-Dickins, P.: Program Evaluation in Language Education. Palgrave Macmillan, New York (2005) Kvale, S.: Interviews: An Introduction to Qualitative Research Interviewing. Sage, Thousand Oaks (1996) Lynch, P.K.: Language Program Evaluation Theory and Practice. Cambridge University Press, Cambridge (1996) Middlebrook, G.C.: Evaluation of student services in ESL programs. In: Pennington, M.C. (ed.) Building Better English Language Programs: Perspectives on Evaluation in ESL, pp. 135–154. NAFSA, Washington (1991) Norris, J.M.: The why (and how) of assessing student learning outcomes in college foreign language programs. Mod. Lang. J. 90, 576–583 (2006) Norris, J.M.: Understanding and improving language education through program evaluation: introduction to the special issue. Lang. Teach. Res. 13(1), 7–13 (2009) Patton, M.: Qualitative Evaluation and Research Methods, 2nd edn. Sage, Newbury Park (1990) Pawan, F., Thomalla, T.G.: Making the invisible visible: A responsive evaluation study of ESL and Spanish language services for immigrants in a small rural county in Indiana. TESOL Q. 39(4), 683–705 (2006) Rivera, G., Matsuzawa, C.: Multiple-language program assessment: learners’ perspectives on first- and second-year college second language programs and their implications for program improvement. Foreign Lang. Ann. 40, 569–583 (2007) Ross, S.J.: A diachronic coherence model for language program evaluation. Lang. Learn. 53(1), 1–33 (2003) Rossi, P., Freeman, H., Lipsey, M.: Evaluation: A Systematic Approach, 6th edn. Sage, Thousand Oaks (1999) Sanders, R.: Redesigning introductory Spanish: increased enrollment, online management, cost reduction, and effects on student learning. Foreign Lang. Ann. 38, 523–532 (2005) Shaw, I., Lishman, J.: Evaluation and Social Work Practice. Sage, London (1999) Shawer, S.F.: Classroom-level teacher professional development and satisfaction: Teachers learn in the context of classroom-level curriculum development. Prof. Dev. Educ. 36(4), 597–620 (2010) Shawer, S.F.: Standardized Assessment and Test Construction Without Anguish: The Complete Step-by-Step Guide to Test Design, Administration, Scoring, Analysis, and Interpretation. Nova Science Publishers, New York (2012)

123

Accreditation and standards-driven program evaluation

2913

Stake, R.E.: Program evaluation particularly responsive evaluation. Multidiscip. Eval. 7(15), 180–201 (2011) Sullivan, J.H.: The importance of program evaluation in collegiate foreign language programs. Mod. Lang. J. 90(4), 590–593 (2006) Sywelem, M., Witte, J.: Higher education accreditation in view of international contemporary attitudes. Contemp. Issues Educ. Res. 2(2), 41–54 (2009) Topkaya, E., Kucuk, O.: An evaluation of 4th and 5th grade English language teaching program. Elementary Educ. Online 9(1), 52–65 (2010)

123