The following editorial and article appeared in the Pennsylvania State University DEOSNEWS on February 28, 2000 (Volume 10, Number 2).

EDITORIAL

As universities are rapidly putting courses and programs into online formats, the issue of discussion between students and faculty and among faculty raises its head. Courses without such interaction amount to little more than self-tutorials or electronic correspondence courses. They lack the added value of (almost) immediate feedback to the students through participation in discussion with their peers, claimed by some to be the best means of adult learning. Carefully crafted questions allow the instructor to monitor student progress in the acquisition and application of course information and concepts.

Many articles and book chapters on online teaching instruct the online faculty person to use questions as a means of starting and fostering discussion among course members. They rarely provide a theoretical basis for the rational development of questions nor tested examples to use as models. The assumption may be that classroom teachers - who are now moving their teaching online - have already mastered this skill.

This month's article not only suggests a theoretical framework for developing such questions but presents the results of research among successful online instructors who share question formats that have engendered and perpetuated discussion. This framework and the examples would be equally valuable when used in face-to-face classrooms.

The first author and principal researcher is a doctoral candidate at the University of South Alabama. Ms. Muilenburg earned her Master's degree in UMBC's Training Systems program where Dr. Berge is Director.

mauri collins
DEOSNEWS EDITOR

******************************

A Framework for Designing Questions for Online Learning

Lin Muilenburg
University of South Alabama
2109 Woodford Court
Mobile, AL 36695
LinM@zebra.net

Zane L. Berge, Ph.D.
UMBC
1000 Hilltop Circle
Baltimore MD 21250
berge@umbc.edu

ABSTRACT

The discussion method is one of the most commonly used pedagogical techniques in the online classroom. Discussion is widely used because it can promote several types of thinking-and certain types of thinking especially those characterized as constructivist, are important in education. Proper attention to the design, facilitation, and maintenance of an online instructional discussion is critical to promote students' constructive thinking. Questioning is a significant instructional design element for the promotion of effective discussion. This article describes a theoretical framework for designing questions for starting online discussion and follow-up questions to maintain the discussion. This framework is placed within a broader context of discussion within a constructivist, online environment. Numerous examples of discussion questions which were gathered from experienced online instructors are presented with the goal of preparing students and teachers to participate effectively in online discussions.

A FRAMEWORK FOR DESIGNING QUESTIONS FOR ONLINE LEARNING
BEGINNING: THE FIRST AUTHOR
When I was faced with the challenge of teaching my first online course, I was both excited and scared. Although I have taught middle school through graduate school, and led adult training and development over the previous 15 years, teaching online was a whole new challenge for me. I had so many questions! One of the most pressing was, "What kind of pedagogical techniques do online teachers use?" So as any self-respecting, lifelong learner would do, I hit the books to look for some answers and ultimately asked experts - online teachers - to share their experiences.
Early in my literature search, I verified my initial hunch that most online courses, whether web-based or not, rely heavily on discussion, if they are to be more than online correspondence courses. In a recent survey of online instructors, forty-one of the forty-two respondents indicated that they used discussion as a teaching method in their online courses (Berge, 1997). While many other pedagogical techniques (see Collins & Berge, 1996) for encouraging online interactions have been identified, such as role play, case study, and debate (Paulsen, 1995), their implementation requires discussion. This is sometimes accomplished within a threaded, computer conferencing system, and sometimes with an electronic discussion group. I decided that learning how to design, facilitate, and manage an online discussion is critical for success in online teaching.

CONTINUING - WITH THE SECOND AUTHOR
While rarely phrased in this way, one of the more important goals of education, and for life-long learners in or out of formal educational settings, is thinking-to use the mind to arrive at a conclusion, make a decision, draw inferences, to reflect, to reason, to solve problems. While it is not the purpose of this article to define or characterize the many nuances of thinking, we will mention several types of thinking below as germane to this article. We will describe a theoretical framework for designing questions to provoke online discussion and follow-up questions to maintain the discussion. This framework is placed within a broader context of discussion within a constructivist, online environment. We will present a selection of discussion questions that were gathered from experienced online instructors are presented with the goal of preparing students and teachers to participate effectively in online discussions.

DISCUSSION PROMOTES THINKING
In a classroom situation, one of the primary - but usually unstated - goals of discussion is to promote thinking. At least four types of thinking are identified in the literature as being promoted by discussion: critical thinking, higher-order thinking, distributed thinking, and constructive thinking.

CRITICAL THINKING. Brookfield (1991) states, "The overarching purpose of discussion is to help learners to explore their experience so that they become more critical thinkers" (p. 192). Critical thinking is a skill that "involves not only knowledge of content but also concept formation and analysis, reasoning and drawing conclusions, recognizing and avoiding contradiction, and other essential cognitive activities" (Scheinin, 1995 cited in Supon, 1998, p. 294).

HIGHER-ORDER THINKING. Higher-order thinking is thinking creatively, critically or in a decision-making or problem solving manner (Sparapani, 1998). Savage (1998) describes higher-order thinking as formulating generalizations from the information learned and then substantiating those generalizations.

DISTRIBUTED THINKING. Distributed thinking or shared cognition can be said to occur in classroom discussion because the connected mental acts of thinking are spread out among a number of different individuals. Higher-order thinking may follow distributed thinking if the new information is internalized, synthesized and then shared with the group (Lipman, 1998).

CONSTRUCTIVE THINKING. Constructive thinking constructs knowledge from personal experience. It involves social interaction and collaboration to share multiple perspectives, and integrating personal experience, personal interpretations of the world and the perspectives of others to create socially constructed meaning (Wilson, Teslow and Osman-Jouchoux, 1995). Constructive thinking includes both critical and creative intellectual processes (Manzo, 1998).

These four types of thinking are related and interconnected during an instructional discussion. As defined above, higher-order thinking subsumes critical thinking, and constructive thinking subsumes distributed thinking and critical thinking. Given the hierarchical nature of these concepts, these types of thinking will be collectively referred to as "constructive thinking" in this paper.

BARRIERS TO CONSTRUCTIVIST THINKING
Manzo (1998) identifies a number of factors that discourage constructive thinking.
1. Schools are set up to transmit existing knowledge. This is in direct conflict with constructive thought.
2. Teachers have not been educated in a climate conducive to constructive thinking, so don't know how to create and promote that atmosphere.
3. The power structure is uncomfortably disrupted for many faculty when students are on equal footing with the instructor.
4. Grading learning is more difficult. Constructive thinking is divergent and therefore difficult to assess with objective tests.
Of these factors, the second may be the most challenging, because desire, and change in beliefs about learning, are not always sufficient to overcome a lack of experience. If you have never experienced constructive teaching methods, how can you do it yourself? Where do you learn to use methods that promote constructive thinking?

PROMOTING CONSTRUCTIVE THINKING IN AN ONLINE DISCUSSION
Haggard (1976, cited in Manzo, 1998, p. 289.) suggests utilizing a four step method to integrate constructive thinking into the curriculum: 1) pose a stimulating question, 2) brainstorm answers to the question, 3) compare ideas, and 4) fuse to the curriculum. In this model, constructive thinking is driven by the instructor's effective questioning technique. The level of student thinking is directly proportional to the level of questions asked. Savage (1988) state:

Research has shown that the greatest majority of a teacher's instructional time is spent asking students questions (Dillon 1982; Feldhusen & Treffinger 1980; USDOE 1986). Seventy to eighty percent of the questions asked require factual recall (USDOE 1980), what Schiever (1991) refers to as "shrinking questions." This in spite of the fact that 80 percent to 90 percent of what students learn through factual questioning is forgotten! Research has also shown that higher-level questions elicit higher cognitive processes (Batson 1981; McKenzie 1972; Taba 1966), and 80 percent to 85 percent of what students learn by such questioning is retained. (p. 291)

From this it would seem that one of the first steps in designing a rich online discussion is to learn to write thought-provoking higher-level questions.

Writing a Good Instructional Question
Hunkins (1972) recommends the following procedure for formulating effective questions:

1) Learn about the cognitive levels of questions (question types)
2) Analyze the educational situation
3) Consider the goals and objectives
4) Consider what question types meet those goals and objectives and the needs of the students
5) Write or select questions
6) Judge the effectiveness of the questions

As indicated in step one above, to write effective questions one must have an understanding of the cognitive levels of questions or the question types. One well-known and widely accepted way to categorize the cognitive levels of questions is Bloom's Taxonomy (Bloom, 1956). Bloom categorizes educational objectives into six categories: knowledge, comprehension, application, analysis, synthesis, and evaluation. There is still argument today as to whether these categories are hierarchical building blocks or just discrete from each other. However, many educators consider the last four categories to represent higher-order thinking. There are many textbooks available that can help in learning to write questions using Bloom's Taxonomy. Still, for the novice, it can be a somewhat cumbersome process.
A simpler framework that can aid in writing constructive questions is Hyman's (1979)
Cognitive Process: Expanded Set of Five Categories
..
A. Definitional 1. Definitions
B. Empirical 2. Facts - Specific or general
3. Relations between facts (comparisons, purposes, explanations, causes, consequences, predictions, or sequences)
C. Evaluative 4. Opinions
5. Justifications of Opinions

In Hyman's categories, higher-order thinking occurs in categories three and five. Knowing a fact (2) is lower-order because it requires only memorization, even though some memorization can be very difficult. Finding the relations between facts (3) is higher-order thinking because one must make comparisons, explain causes, make predictions, etc. Making an evaluation based only on one's own opinion (4) is considered lower-order thinking, whereas justifying that opinion is considered higher-order thinking. Using this simpler framework, constructive questions would be those that ask learners to express the relationships among things, or ask them to justify or support their opinions.

FUNCTIONS OF QUESTIONS
Hunkins (1972) identifies another categorization for questions according to "question function." The two functions that are important to this discussion are centering (questions which promote convergent thought) and expanding (questions which promote divergent thought). Experienced online instructors have found that expanding or divergent questions tend to produce the richest online discussions.

SAMPLE QUESTIONS
Even with guidelines for writing questions to promote critical thinking, it can be difficult for instructors and students to come up with good questions. Unlike public scholarly discussion groups used for professional development, most online classroom discussions are private or password-protected, and it is hard to find and access these online discussions to see effective questions in action.

RESEARCH METHODOLOGY AND RESULTS - FIRST AUTHOR

To gather information on the types of questions instructors use, I sent requests for examples of questions to two, preeminent professional development mailing lists in the instructional design and distance education fields: ITFORUM and DEOS-L. Additionally, individual e-mails were sent to 36 instructors identified as having extensive online teaching experience. Approximately two dozen respondents returned fifty sample questions in this non-scientific survey. These examples helped form the bases for the examples here. In general, questions for instructional purposes can be grouped into the following categories (with examples) (Borich, 1996):

Interest-getting and attention-getting. "If you awakened in the year 2399, what is the first thing you would notice?"

Diagnosing and checking. "Does anyone know Senge's five principles of a learning organization?"

Recall of specific facts or information. "Who can name the main characters in Moby Dick?"

Managerial. "Did you request an extension on the assignment due date?

Structure and redirect learning. " Now that we have discussed the advantages of, and limitations to, formative evaluation, who can do the same for summative evaluation?"

Allow expression of affect. "How did you feel about our online guest's list of ten things trainers do to shoot themselves in the foot?"

Encourage higher level thought processes. "Considering what you have read, and what was discussed in the posts this past week, can you summarize all the ways there are to overcome obstacles to effective teamwork?"
Given that constructivist methods focus on encouraging higher order thinking, it is this category of encouraging higher level thought processes, that is of particular focus for the remainder of this article.

ENCOURAGING HIGHER LEVEL THOUGHT PROCESSES

Examples of Literature-Based (read and think) Questions

The majority of the following sample questions selected for inclusion here have to do with assigned-readings, or literature-based examples; these are listed separately. Within the context of literature-based examples, instructors formulated questions using a wide variety of approaches.

Focus on Main Topic or Issue: "John Covaleski's piece really challenges the whole notion of conducting systemic reform. His central argument is that mediocrity is systemic; excellence cannot be. While no one would dispute that there are examples of excellence in particular classrooms, as a whole the educational system seems to tend toward minimal competence rather than excellence. If it is the case that the system itself, like the universe tending toward chaos, works against broad spread of excellence, we must ask the question: what then is (or should be) the goal of reform efforts?"

Series Of Divergent Questions with Expanding Follow-Up: The following questions are the first two in a series of seven specific questions on one reading assignment. "1) Look at the first graph on prison population explosion. Another part of the article says that prison population has tripled since 1989. Do you think that the number of crimes has tripled since 1989? Do you think that the types of crimes committed have become more violent in nature? 2) In the last section of part one prisoners discuss the conditions in maximum security. What is the psychological effect of depriving someone of things like soap and deodorant?"
This series of divergent questions was followed-up with a writing assignment that asked for expansion.
"Answer Ted Koppel's question 'Is this where we want to go?' Your answer should include where you think the current prison system is leading to, and why or why not this is where we want to go. If this system is not leading us to where we want to go, give alternatives that might get us back on track."
Another example is: "Of the restructuring issues we'll deal with in this course, probably none is as controversial as the issue of choice. So, naturally, we're starting with that one. Choice is an issue that can be understood in a variety of ways and in a variety of contexts. How freely shall we allow the schools of a district to vary? Teachers within a school? For that matter, how freely shall we allow the districts of a state to vary? How do we (can we?) meet the education needs of ALL students within a given school? Is the traditional school doing a very good job of that now? Where is the student's choice? The parents'? What does this business of choice mean to you?"

Evaluative: Who was more convincing: Coulson with his ideas of education as goods to be freely sought for and provided, or Covaleski with his idea of the common good?

Tie in to current events: "_Patrick Shields helps us consider the place of parents and community in the school reform movement. Anyone reading the paper lately? We have a home-grown parent and community situation right here in dear old West Lafayette. How can Shield's framework help us understand what's happening on the West Side?"

Quote Contrasting Views: "Here are two divergent perspectives on the role of technology in education. The first is from Howard Mehlinger, former Dean of Education and now Director of the Center for Excellence in Education at Indiana University. (I just saw Howard at a conference in Washington last week, and it made me think of this.) The second is from Steven Jobs, founder of Apple Computer, who seems now to be trying to revitalize the fortunes of that company.
'Technology is the key to a new school vision and a revised mission. It is not that we should worship technology anymore than astronauts worship rocket boosters. Technology is important only to the degree that it helps us get where we want to go. If we have a vision of schooling that calls for students to work at their capacity, at their own pace, at tasks they find to be challenging and enjoyable, we are likely to be successful only if we take advantage of the opportunities afforded us by new technologies.' - Howard Mehlinger, 1995, School Reform in the Information Age
'I used to think that technology could help education. I've probably spearheaded giving away more computer equipment to schools than anybody else on the planet. But I've had to come to the inevitable conclusion that the problem is not one that technology can hope to solve. What's wrong with education cannot be fixed with technology. No amount of technology will make a dent.' - Steven Jobs, February 1996 issue of Wired magazine
"Is one or the other of these guys right? Is technology a key to educational reform, something that can't make any difference at all, or maybe something else altogether? What's your take on this issue?"

Building on Classmates Posts: Post #1: "Identify three major social changes that occurred from 1865-1910. Explain each of these by detailing what happened to cause this change, the time span it took for the change to occur, and the way the change was expressed in the literature we will be reading."
Post #2: "Compare the changes you identified with the changes that your classmates identified. Discuss the merits of the changes your classmates identified that you did not. Reflect and see if you have changed or modified your original three ideas."
Post #3: "You will not be able to complete post #3 until you read the literature for the unit. When you have, you will focus your answers author (sic) and on his/her piece(s) of literature that was assigned to your group. Using the information about your author and his/her piece(s) of literature your group researched, show how this piece of literature reflects one of the changes you or a classmate identified."

Example Questions from Other Than Assigned-Reading

Scenarios, critical incidents or problems: "
Describe a scenario and ask questions such as: If you were consulting in this situation, how would you approach it? What might some of your recommendations be? What data would you want to gather, if any, before making recommendations?"

Case study: One online teacher sent a lengthy case that described in detail the interactions between a Vice President of Loans (Sam, the boss) and a commercial lending specialist who is up for promotion (Nancy, the worker). Under the guise of discussing her potential for promotion, Sam pressures Nancy into going out to dinner and then asks her to go to his place. Nancy thinks this might be sexual harassment but is concerned that turning Sam down will jeopardize her promotion. Final Questions: "What do you think Nancy and Sam's personality, values, ethics, attitudes, beliefs, and maturity levels are? If you were Nancy what would you say and do?"

Controversial or provocative questions: "Perhaps a better idea is to pose controversies or questions which have no easy answers, or which have many possible answers with unique advantages or disadvantages. If students are in a position to add their experiences and insights as practitioners then this becomes a rich discussion," stated one online teacher. "The best result though, was when I posted a few paragraphs asking students to comment on the thin line between helping someone with a computer programming assignment and cheating. A great discussion ensued."

Find and critique Websites: Another online teacher commented, "I might ask which Web site they found that has the greatest potential value for helping their students achieve a specific objective. They then give a URL and a brief reason why they like it and where it might be used. Others read it and react to its value to them before suggesting their own URL choice."

Role Play: Another teacher said, "I would tell the class that they are attending a Board of Directors' meeting asking for funding for new computers. Several board members are not in favor of spending money on technology. I would assign class members roles as the librarian making the request and others as board members against the money."

CONTINUING - WITH THE SECOND AUTHOR

MANAGE THE DISCUSSION IN PROCESS
Once the discussion has started and students are beginning to post responses to the question(s) provided, what should be done to facilitate this discussion? Feedback is important, especially individually given feedback in private email to encourage new-comers to contribute to the discussion and to welcome participants. These private notes can explain how individual students might respond in a clearer manner, suggest a need to spell check, or point out a need to watch the tone or attitude that has been taken (Burke, 1999).
The types of questions used to promote effective, ongoing discussions are somewhat different than those used to start discussion. These probing questions could be the topic of another article, but to summarize, Savage (1998) suggests probing question such as:
Ö What reasons do you have for saying that?
Ö Why do you agree (or disagree) on that point?
Ö How are you defining the term that you just used?
Ö What do you mean by that expression?
Ö Is what you are saying here consistent with what you said before?
Ö Could you clarify that remark?
Ö When you say that, what is implied by your remarks?
Ö What could follow from what you just said?
Ö Is it possible you and he are contradicting each other?
Ö Are you sure you're not contradicting yourself?
Ö What alternatives are there to such a formulation?

Jumping In
How do instructors know when to jump into the ongoing discussion with their comments? If things are going well, the best action to take is no action! It is best not to interfere. Resist the temptation, if it exists, to post a public reponse to the question until conversation is waning. The best course of action then is to either summarize the key points to end discussion on the topic or ask some prompting questions to recharge the discussion. Even more so than in-person, online discussion is usually stifled by a well articulated, (especially lengthy) post that gives the answer from the instructor. It tends to be perceived as "the final word" by students-acting just as a summary should for the topic.

CONCLUSIONS
The discussion method is the most popular pedagogical technique used in the online classroom. To use the discussion method effectively, it is critical to understand how to design and maintain an online discussion. When a variety of higher-order, expanding questions are used to initiate discussion, and probing follow-up questions are employed, the discussion method can provide a forum to enhance constructive thinking. Learners can be exposed to multiple perspectives and view issues from the perspective of others. Students may be forced to examine the assumptions which underlie their values, beliefs and actions (Brookfield, 1991). Unstructured problems and the complex and ambiguous nature of many topics can be examined. In a constructivist learning environment, the instructor always needs to keep in mind that when facilitating online discussion, asking the right questions is almost always more important than giving the right answers.



REFERENCES
Batson, A. D. (1981). Questioning: A reading/thinking foundation for the gifted. Paper presented at the Annual Meeting of the Southwest Regional Conference of the International Reading Association, San Antonio, Texas, 29-31 Jan. (ERIC Document Reproduction Service No. ED 201 999.)

Beaudin, B. P. (1999). Keeping online asynchronous discussions on topic. Journal of Asynchronous Learning Networks, 3(2). [Electronic Journal]. Available: http://www.aln.org/alnweb/journal/Vol3_issue2/beaudin.htm [1999, 99/11/12].

Berge, Z., L. (Accepted for publication). Online teaching: Purpose and methods. Electronic Journal of Communication.

Berge, Z.L. (1997). Characteristics of online teaching in post-secondary, formal education. Educational Technology, 37(3), 35-47.

Bloom, B. (1956). Taxonomy of educational objectives. New York, NY: David McKay Company, Inc.

Borich, G.D. (1996). Effective teaching methods. (3rd Ed.). Englewood Cliffs, NJ: Merrill.

Brookfield, S. D. (1991). Discussion. In M. W. Galbraith (Ed.), Adult Learning Methods (pp. 187-204). Malabar, Florida: Krieger Publishing Company.

Burke, A. N. (1999). Helpful files for online teaching, [Website]. Burke, A. Nadine. Available: http://www.delta.edu/~anburke/mentor/ [1999, 99/11/16].

Collins, M., & Berge, Z. L. (1996, 99/3). The moderators' homepage: Resources for moderators and facilitators of online discussion, [website]. Mauri Collins. Available: http://www.emoderators.com/moderators.shtml.

Dillon, J.T. (1982). The multi-disciplinary study of questioning. Journal of Educational Psychology. 74(2): 147-65.
Eisley, M. E. (1999). Guidelines for conducting instructional discussions on a computer conference, [DEOSNEWS Electronic Serial]. Available: http://www.ed.psu.edu/ACSDE/deosnews2.1.htm [1999, 99/11/12].

Feldhusen, J. F., and Treffinger, D. J. (1980). Creative thinking and problem solving in gifted education. Dubuque, Iowa: Kendall/Hunt.

Gage, N.L. and Berliner, D.C. (1992). Educational psychology. (5th Ed.). Princeton, NJ: Houghton Mifflin Company.

Haggard, M.R. (1976). Creative thinking-reading activities (CT-RA) as a means for improving comprehension. Doctoral dissertation, University of Missouri - Kansas City.

Hunkins, F. P. (1972). Questioning strategies and techniques. Boston, MA: Allyn & Bacon, Inc.

Hyman, R. T. (1979). Strategic questioning. Englewood Cliffs, NJ: Prentice-Hall, Inc.

Jacobsen, D., Eggen, P., and Kauchak, D. (1993). Methods for teaching: A skills approach. (4th Ed.). Upper Saddle River, NJ: Merrill.

Lipman, M. (1998). Teaching students to think reasonably: Some findings of the philosophy for children program. The Clearing House, 71(5), 277-280.

Manzo, A. V. (1998). Teaching for creative outcomes: Why we don't, how we all can. The Clearing House, 71(5), 287-290.

McKenzie, G. R. (1972). Some effects of frequent quizzes on inferential thinking. American Educational Research Journal 9(2): 231-40.

Paulson, M. F. (1995). The online report on pedagogical techniques for computer-mediated communication, [Online Report]. Available: http://www.emoderators.com/index.html [1999, 99/10/15].

Savage, L. B. (1998). Eliciting critical thinking skills through questioning. The Clearing House, 71(5), 291-293.

Scheinin, P.M. (1995). Improving Thinking Skills. Available: http://www.helsinki.fi/scheinin/abs6.html [97/06/27].

Schiever, S.W. (1991). A comprehension approach to teaching and thinking. Boston: Allyn and Bacon.

Sparapani, E. F. (1998). Encouraging thinking in high school and middle school: constraints and possibilities. The Clearing House, 71(5), 274-276.

Supon, V. (1998). Penetrating the barriers to teaching higher thinking. The Clearing House, 71(5), 294-296.

Taba, H. (1966). Teaching strategies and cognitive functioning in elementary school children. Cooperative Research Project No. 2404. San Francisco State College.

USDOE (U. S. Department of Education) (1980). What works: research about teaching and learning. Washington, D.C.: USDOE.

Wilson, B., Teslow, J., & Osman-Jouchoux, R. (1995). The impact of constructivism (and postmodernism) on ID fundamentals. In B. B. Seels (Ed.), Instructional Design Fundamentals: A Review and Reconsideration (pp. 137-157). Englewood Cliffs, NJ: Educational Technology Publications.

---------------------------------------------

END NOTE
A special note of appreciation to those who sent examples privately: Ron Barnette, Sheryl Burgstahler, Nadine Burke, Steve Draper, Gene Ortiz, Guy Hancock, Greg Kearsley, Claudine Keenan, Crawford Killian, Judi Kirkpatrick, William Klemm, James Lehman, Ralph Lewis, Lucy MacDonald, Cecilia Manrique, Al Mizell, Som Naidu, Steven Naramontas, Sarah Phinney, Roy Reeber, Judy Scholl, Molly Burgess Smith, Jan Strever, Danney Ursery, and Lynn White.