research question generator science

Research Question Generator for Free

If you’re looking for the best research question generator, you’re in the right place. Get a list of ideas for your essay, research paper, or any other project with this online tool.

• 🎓 How to Use the Tool
• 🤔 What Is a Research Question?
• 😺 Research Question Examples
• 👣 Steps to Making a Research Question

📝 Research Question Maker: the Benefits

🔗 references, 🎓 research question generator: how to use it.

Research can’t be done without a clear purpose, an intention behind it.

This intention is usually reflected in a research question, which indicates how you approach your study topic.

If you’re unsure how to write a good research question or are new to this process, you’ll surely benefit from our free online tool. All you need is:

• Indicate your search term or title
• Stipulate the subject or academic area
• Press “Generate questions”
• Choose a suitable research question from the generated list.

As you can see, this is the best research question generator requiring minimal input for smart question formulation. Try it out to see how simple the process is.

🤔 Why Make an Inquiry Question?

A research question is a question that you formulate for your scientific inquiry . It is a question that sets the scope for your study and determines how you will approach the identified problem, gap, or issue.

Questions can be descriptive , meaning they aim to describe or measure a subject of the researcher's interest.

Otherwise, they can be exploratory , focusing on the under-researched areas and aiming to expand the existing research evidence on the topic.

If there's enough knowledge about the subject, and you want to dig deeper into the existing trends and relationships, you can also use an explanatory research question.

What Makes a Strong Research Question?

The strength of your formulated research question determines the quality of your research, whether it’s a short argumentative essay or an extensive research paper . So, you should review the quality of your question before conducting the full-scale study.

Its parameters of quality are as follows:

• Clarity . The question should be specific about the focus of your inquiry.
• Complexity . It should not be self-obvious or primitively answered with a “yes” or “no” variant.
• Focus . The question should match the size and type of your academic assignment.
• Conciseness . It should be brief and understandable.
• Debatability . There should be more than one potential answer to the question.

😺 Research Question Examples: Good & Not So Good

Here are some examples to illustrate what we mean by quality criteria and how you can ensure that your question meets them.

Lack of Clarity

The bad example is too general and does not clearly estimate what effect you want to analyze or what aspect of video gaming you're interested in. A much better variant is in the right column.

Look at some other research question examples that are clear enough:

• Terrorism: what is it and how to counter it?
• Sex trafficking: why do we have to address it?
• Palliative care: what constitutes the best technique for technicians communication with patients and families?
• How do vacuum cleaners work?
• What does it mean to age well?

Lack of Focus

The bad example is not focused, as it doesn’t specify what benefits you want to identify and in what context the uniform is approached. A more effective variant is in the right column.

Look at some other research question examples that are focused enough:

• How are biochemical conditions and brain activity linked to crime?
• World wars and national conflicts: what were the reasons?
• Why does crime exist in society?
• Decolonization in Canada: what does decolonization mean?

The bad example is too simplistic and doesn’t focus on the aspects of help that dogs can give to their owners. A more effective variant is in the right column.

Look at some other research question examples that are complex enough:

• How is resource scarcity impacting the chocolate industry?
• What should the Brazilian government do about reducing Amazon’s deforestation?
• Why is a collaborative approach vital during a pandemic?
• What impact has COVID-19 had on the economy?
• How to teach handwriting effectively?

Lack of Debatability

The problem of diabetes is well-known and doesn’t cause any doubts. So, you should add debatability to the discussed issue.

Look at some other research question examples that are debatable enough:

• Online vs. print journalism: what is more beneficial?
• Why will artificial intelligence not replace human in near future?
• What are the differences between art and design?
• Crime TV: how is criminality represented on television?

The question in the left column is too long and ambiguous, making the readers lose focus. You can shorten it without losing the essence.

Look at some other research question examples that are concise enough:

• What is the best way to address obesity in the US?
• Doctoral degree in nursing: why is it important?
• What are the benefits of X-rays in medicine?
• To what extent do emotions influence moral judgment?
• Why did the Industrial Revolution happen in England?

👣 Steps to Generate Research Questions

Now, it’s time to get down from science to practice. Here is a tried-and-tested algorithm for killer research question generation.

• Pick a topic . Once you get a writing assignment, it’s time to find an appropriate topic first . You can’t formulate a thesis statement or research question if you know nothing about your subject, so it's time to narrow your scope and find out as much as possible about the upcoming task.
• Research the topic . After you’re brainstormed several topic options, you should do some research. This stage takes the guesswork out of the academic process, allowing you to discover what scholars and other respected people think about your subject.
• Clarify who your audience is . Think about who will read your piece. Will it be the professor, your classmates, or the general audience consisting of laypersons? Ensure the research question sounds competent enough for a professor and understandable enough for laypeople.
• Approach the subject critically . With a well-articulated topic at hand, you should start asking the "why's" and "how's" about it. Look at the subject as a kid; don't limit your curiosity. You're sure to arrive at some interesting topics to reveal the hidden sides of the chosen issue.
• Evaluate the questions . Now that you have a couple of questions about your topic, evaluate them in terms of research value. Are all of them clear and focused? Will answering all of them take time and research, or is the answer already on the surface? By assessing each option you’ve formulated, you’re sure to choose one leader and use it as your main research question for the scientific study.

Thank you for reading this article! If you need to quickly formulate a thesis statement, consider using our free thesis maker .

❓ Research Questions Generator FAQ

Updated: Oct 25th, 2023

• Developing research questions - Library - Monash University
• Formulation of Research Question – Stepwise Approach - PMC
• Examples of Good and Bad Research Questions
• How To Write a Research Question: Steps and Examples
• Narrowing a Topic and Developing a Research Question
• Free Essays
• Writing Tools
• Lit. Guides
• Donate a Paper
• Referencing Guides
• Free Textbooks
• Tongue Twisters
• Job Openings
• Expert Application
• Video Contest
• Writing Scholarship
• Discount Codes
• IvyPanda Shop
• Terms and Conditions
• Privacy Policy
• Cookies Policy
• Copyright Principles
• DMCA Request
• Service Notice

With our question generator, you can get a unique research question for your assignment, be it an essay, research, proposal, or speech. In a couple of clicks, our tool will make a perfect question for you to ease the process of writing. Try our generator to write the best work possible.

Research Question Generator: Best Tool for Students

Stuck formulating a research question? Try the tool we’ve made! With our research question generator, you’ll get a list of ideas for an academic assignment of any level. All you need to do is add the keywords you’re interested in, push the button, and enjoy the result!

Now, here comes your inspiration 😃

Please try again with some different keywords.

Why Use Research Question Generator?

The choice of research topic is a vital step in the process of any academic task completion. Whether you’re working on a small essay or a large dissertation, your topic will make it fail or fly. The best way to cope with the naming task and proceed to the writing part is to use our free online tool for title generation. Its benefits are indisputable.

• The tool generates research questions, not just topics
• It makes questions focused on your field of interest
• It’s free and quick in use

Research Question Generator: How to Use

Using our research question generator tool, you won’t need to crack your brains over this part of the writing assignment anymore. All you need to do is:

• Insert your study topic of interest in the relevant tab
• Choose a subject and click “Generate topics”
• Grab one of the offered options on the list

The results will be preliminary; you should use them as an initial reference point and refine them further for a workable, correctly formulated research question.

Research Questions: Types & Examples

Depending on your type of study (quantitative vs. qualitative), you might need to formulate different research question types. For instance, a typical quantitative research project would need a quantitative research question, which can be created with the following formula:

Variable(s) + object that possesses that variable + socio-demographic characteristics

You can choose among three quantitative research question types: descriptive, comparative, and relationship-based. Let's consider each type in more detail to clarify the practical side of question formulation.

Descriptive

As its name suggests, a descriptive research question inquires about the number, frequency, or intensity of something and aims to describe a quantitative issue. Some examples include:

• How often do people download personal finance apps in 2022?
• How regularly do Americans go on holidays abroad?
• How many subscriptions for paid learning resources do UK students make a year?

Comparative

Comparative research questions presuppose comparing and contrasting things within a research study. You should pick two or more objects, select a criterion for comparison, and discuss it in detail. Here are good examples:

• What is the difference in calorie intake between Japanese and American preschoolers?
• Does male and female social media use duration per day differ in the USA?
• What are the attitudes of Baby Boomers versus Millennials to freelance work?

Relationship-based

Relationship-based research is a bit more complex, so you'll need extra work to formulate a good research question. Here, you should single out:

• The independent variable
• The dependent variable
• The socio-demographics of your population of interest

Let’s illustrate how it works:

• How does the socio-economic status affect schoolchildren’s dropout rates in the UK?
• What is the relationship between screen time and obesity among American preschoolers?

Research Question Maker FAQ

In a nutshell, a research question is the one you set to answer by performing a specific academic study. Thus, for instance, if your research question is, “How did global warming affect bird migration in California?," you will study bird migration patterns concerning global warming dynamics.

You should think about the population affected by your topic, the specific aspect of your concern, and the timing/historical period you want to study. It’s also necessary to specify the location – a specific country, company, industry sector, the whole world, etc.

A great, effective research question should answer the "who, what, when, where" questions. In other words, you should define the subject of interest, the issue of your concern related to that subject, the timeframe, and the location of your study.

If you don’t know how to write a compelling research question, use our automated tool to complete the task in seconds. You only need to insert your subject of interest, and smart algorithms will do the rest, presenting a set of workable, interesting question suggestions.

Research Question Generator for Students

Our online topic question generator is a free tool that creates topic questions in no time. It can easily make an endless list of random research questions based on your query.

Can't decide on the topic question for your project? Check out our free topic question generator and get a suitable research question in 3 steps!

Please try again with some different keywords.

• 👉 Why Use Our Tool?

💡 What Is a Topic Question?

✒️ how to write a research question.

• 📜 Research Question Example

🔗 References

👉 why use our topic question generator.

Our research topic question generator is worth using for several reasons:

• It saves you time. You can develop many ideas and formulate research questions for all of them within seconds.
• It is entirely free. Our tool doesn’t have any limits, probation periods, or subscription plans. Use it as much as you want and don’t pay a cent.
• It is download- and registration-free. Use it in any browser from any device. No applications are needed. You also don’t have to submit any personal data.
• It’s easy to use. You can see an explanation for every step next to each field you need to fill in.
• You can easily check yourself. Spend a couple of seconds to check your research question on logic and coherence.

A research topic question is a question you aim to answer while researching and writing your paper. It states the matter you study and the hypothesis you will prove or disprove. This question shares your assumptions and goals, giving your readers a basic understanding of your paper’s content.

It also helps you focus while researching and gives your research scope and limitations. Of course, your research question needs to be relevant to your study subject and attractive to you. Any paper will lack an objective and specificity without an adequately stated research question.

Research Topic Vs. Research Topic Question

‘Research topic’ and ‘research question’ are different concepts that are often confused.

Research Question Types: Quantitative and Qualitative

Another essential differentiation to know – there are quantitative and qualitative research questions.

• Quantitative research questions are more specific and number-oriented . They seek clear answers such as “yes” or “no,” a number, or another straightforward solution. Example: How many senior high school students in New York failed to achieve the desired SAT scores due to stress factors?
• Qualitative research questions can be broader and more flexible. They seek an explanation of phenomena rather than a short answer. Example: What is the role of stress factors in the academic performance of high school senior students who reside in New York?

Now let’s get to know how to create your own research question. This skill will help you structure your papers more efficiently.

Step 1: Choose Your Research Topic

If you’ve already received general guidelines from your instructor, find a specific area of knowledge that interests you. It shouldn’t be too broad or too narrow. You can divide it into sub-topics and note them. Discuss your topic with someone or brainstorm to get more ideas. You can write down all your thoughts and extract potential issues from this paragraph or text.

Step 2: Research

After you’ve chosen a topic, do preliminary research . Search for keywords relevant to your topics to see what current discussions are in the scientific community. It will be easier for you to cross out those ideas that are already researched too well. In addition, you might spot some knowledge gaps that you can later fill in. We recommend avoiding poorly researched areas unless you are confident you can rely solely on the data you gather.

Step 3: Narrow Your Topic

At this stage, you already have some knowledge about the matter. You can tell good ideas from bad ones and formulate a couple of research questions. Leave only the best options that you actually want to proceed with. You can create several draft variations of your top picks and research them again. Depending on the results you get, you can leave the best alternatives for the next step.

Step 4: Evaluate What You’ve Got

Evaluate your topics by these criteria:

• Clarity . Check if there are any vague details and consider adjusting them.
• Focus . Your research matter should be unambiguous , without other interpretations.
• Complexity . A good topic research question shouldn’t be too difficult or too easy.
• Ethics . Your ideas and word choice shouldn’t be prejudiced or offensive.
• Relevance . Your hypothesis and research question should correspond with current discussions.
• Feasibility . Make sure you can conduct the research that will answer your question.

Step 5: Edit Your Research Question

Now you can create the final version of your research question. Use our tool to compare your interpretation with the one produced by artificial intelligence. Though you might change it based on your findings, you must create a perfect statement now. You need to make it as narrow as possible. If you don’t know how to make it more specific, leave it till you get the first research results.

📜 Research Question Generator: Examples

Compare a good and bad research question to understand the importance of following all rules:

Thank you for reading till the end. We hope you found the information and tool useful for your studies. Don’t forget to share it with your peers, and good luck with your paper!

Updated: May 17th, 2024

• The Writing Center | How to Write a Research Question | Research Based Writing
• How to Write a Research Question: Types, Steps, and Examples | Research.com
• Pick a Topic & Develop a Research Question – CSI Library at CUNY College of Staten Island Library

Quantitative Research Question Generator

A research question is the core of any academic paper. Yet, the formulation of a solid quantitative research question can be a challenging task for students. That’s why the NerdyRoo team created an outstanding tool that will become your ultimate academic assistant.

🚀 Why Use Our Generator?

🔎 what is a quantitative research question.

• ✍️ Writing Steps
• ✨ Question Examples

🔗 References

Doubting whether our quantitative research question generator is worth using? It is! Our tool has many benefits:

• It’s entirely free
• It’s accessible online and without registration
• It’s easy to use
• It saves your time
• It boosts productivity
• It instantly generates a high-quality quantitative research question.

Quantitative questions are close-ended questions used for gathering numerical data. Such questions help to collect responses from big samples and, relying on the findings, make data-driven conclusions. A research question is essential to any quantitative research paper because it presents the topic and the paper's aim.

Quantitative research questions always contain variables : things that are being studied. It's crucial to ensure your variables are attainable and measurable. For example, you cannot measure organizational change and emergency response, but you can determine the frequency of organizational change and emergency response score.

Types of Quantitative Research Questions

Do you know that there are 3 types of quantitative research questions? Take a look at them and decide which type is the most suitable for your paper.

Quantitative vs. Qualitative Research Questions

Many students confuse quantitative and qualitative research . Despite having similar-sounding names, they're very different:

• Quantitative questions are aimed at collecting raw numerical data.
• Qualitative questions have an answer expressed in words. They also allow getting respondents’ personal perspectives on a research topic.

Let’s examine the main differences between qualitative and quantitative research :

✍️ How to Write a Quantitative Research Question

Want to craft an outstanding quantitative research question? We know how to help you! Follow the 5 steps below and get a flawless result:

1. Choose the type of research question.

Decide whether you need a descriptive, comparative, or relationship-based quantitative research question. How your question starts will also depend on the type.

2. Identify the variables.

See how many variables you have. Don't forget to distinguish between dependent and independent ones .

3. Set the groups.

Your study will focus on one or more groups. For example, you might be interested in social media use among Gen-Z Americans, male Millennials, LGBTQ+ people, or any other demographic.

4. Specify the context.

Include any words that will make your research question more specific, such as "per month," "the most significant," or "daily."

5. Compose the research question.

Combine all the elements from the previous steps. Use our quantitative research question generator to ensure the highest result quality!

✨ Quantitative Research Question Examples

Now, let's look at some well-formulated quantitative research questions with explanations for variables and groups.

Thanks for visiting our webpage! Good luck with your quantitative research. Use our online tool and share it with your friends!

❓ Quantitative Research Questions FAQs

❓ what is an example of a quantitative research question.

A quantitative research question might be the following: "What is the relationship between website user-friendliness and customer purchase intention among male and female consumers of age 25 to 30?" Another example would be: "What percentage of Bachelor's graduates acquire a Master's degree?"

❓ What are the quantitative questions?

Quantitative questions are close-ended questions used for collecting numerical data. Such questions help gather responses from big samples and trace patterns in the selected study area. Relying on the finding of quantitative research, the researcher can make solid decisions.

❓ How do you write a quantitative research question?

• Identify the variables.
• Decide on the focus groups.
• Specify the context.

To ensure the best result, use our online generator. It will create a flawless research question for free in a couple of seconds!

❓ What questions does quantitative research answer?

Quantitative research answers any kind of question involving numerical data analysis. For example, it may help to determine the interdependence of variables, examine current trends within the industry, and even create forecasts.

• Developing Your Research Questions: Nova Southeastern University
• How to Write a Research Question: George Mason University
• Quantitative Methods: University of Southern California
• Research Question Overview: North Central University

Create Key Questions to Guide Your Studies

AI Generators in Science and Research

🧪🔍 Formulate relevant questions to guide your scientific work. Define your research objective precisely!

Provide additional feedback

The quality of a study is often based on the relevance of its initial questions. In the vast world of research, asking the right questions is essential to obtaining meaningful, impactful answers.

📝 Questions That Matter

Our generator helps you formulate pertinent, targeted questions to give a clear direction to your research.

💡 Suitable for All Subjects

Whether it's life sciences, astrophysics or sociology We cover a wide range of fields to meet your needs.

🌐 Set your course

With thoughtful questions, optimize reach and efficiency of your research within the scientific community.

Similar publications :

Popular generators:

Create Limitless with Generator AI

Immerse yourself in a world where every idea is instantly transformed into reality. Generator AI brings your boldest visions to life in the blink of an eye.

Quantitative Research Question Generator

Wondering how to come up with a fascinating research question? This tool is for you.

• 📝 Qualitative Research Definition
• ✍️ Creating a Qualitative Research Question
• 👼 Useful Tips

🔗 References

What does the qualitative research question generator do.

Our qualitative research question generator creates science topics for any studying field.

You just need to:

• Add the necessary information into the textboxes;
• Click “Generate”;
• Look at the examples if necessary.

You will see different research areas, from broad to narrow ones. If you didn't find the ones you like, click "see more."

Don't worry if it's your first qualitative research experience. We will teach you how to formulate research questions and get the most out of this tool.

📝 What Is Qualitative Research?

First, let's define the difference between qualitative and quantitative research. Qualitative research focuses on meaning and experience. Quantitative research aims to provide empirical evidence to prove a hypothesis.

Qualitative research is an inquiry information gathering method used to study social phenomena. In most cases, qualitative research relies on interviews, observations, focus groups , and case studies. The main goal of a researcher is to gain deep insights instead of focusing on numbers and statistics. Qualitative research is popular in psychology, anthropology, sociology, political science, education studies, and social work.

Take a look at the 6 characteristics of qualitative research:

• It relies on a real-world setting.
• It uses multiple data collection methods .
• It depends on researchers' data more than on prior research.
• It considers participants' understanding of phenomena.
• It uses complex reasoning.
• It has a flexible course of study.

What Is a Qualitative Research Question?

Qualitative research question tries to understand the meaning people tie to a concept. It seeks to explore the phenomena rather than explain them. For that reason, a qualitative research question is usually more general.

How to write a good qualitative research question? Make sure it meets the requirements below:

• Broadness. Qualitative research questions need to be broad and specific enough.
• Relevance. A topic should interest your audience and follow the given instructions.
• Applicability. The results of your research should have scientific value and implementation opportunities.
• Clarity. Your readers need to know what your paper is about after seeing the research question.
• Flexibility. The scenario of your research can change while you conduct it. That is why qualitative research questions are often open-ended.

✍️ How to Make a Qualitative Research Question?

Now we will explain how to choose and create an excellent research title. Just make sure you follow these three steps:

• Brainstorm your ideas. Take a piece of paper or use any digital device to note your thoughts. Based on the task, make a list of as many research topics as you can.
• Choose the topics that are perfect at this stage. You can start with your interests and the areas already familiar to you. Use your old papers. In case you have noticed any knowledge gaps while in them, add these topics too. Think about how your work can contribute to the chosen science field.
• Conduct a preliminary literature review on your topics. It will help to eliminate the ones that are already studied well.
• Define the purpose of your research. For example, you want to understand what rewards and perks are the most stimulating for employees. Discuss your ideas with your instructor. If you have an opportunity to choose a supervisor, find a professor with experience within your area of interest.
• Choose one topic and think of a research question . If you discuss your research in class, listen to what your peers say about it. It can give you new ideas and insights on the topic.
• After choosing your topic, research the area more thoroughly. You might rely on your supervisor's instructions at this stage. Take notes on your findings and think about how you can use them. Think of the questions you had after reading the materials and how you can answer them.
• Formulate your research question. Make the question as concise as possible. If you find it challenging to put your thoughts into one sentence, write a small paragraph. Then you can shorten it and form a question.
• Create 1-3 sub-questions in addition to your main qualitative research question. They should highlight the purpose of your research and give more information about your work.

✨ Benefits of Qualitative Research Question Generator

Knowing how to create a qualitative research topic is excellent. But imagine what you can do if you combine your knowledge with artificial intelligence.

Here's how students can benefit from using our tool:

• It saves time. The tool generates an infinite list of topics in one second. You just need to choose the ones that appeal to you.
• It is 100% free. No registration, subscription, or donation is required.
• It has no limits. Use this tool as many times as needed.
• It is easy to use. Type your research keywords into the search bar and click "Search topic."
• It values your privacy. You don't need to disclose any personal data to use this tool.

👼 Tips for Writing a Qualitative Research Question

Here are some nice bonus tips for you:

• Use qualitative verbs such as "describe", "understand", "discover".
• Avoid quantitative research-related verbs such as "effect", "influence", "relate".
• Be sure that it is possible to answer the question fully.
• Don't be afraid to adjust your research question if your research leads to biased results.
• Check if your research question and information-gathering methods are ethical.
• Make sure your research question doesn't contradict your purpose statement.
• Rely on research findings rather than on your predictions.

❓ How do you write a qualitative research question?

Brainstorm your ideas and highlight the best ones. To formulate a research question, use qualitative words and broad ideas. Create a paragraph to narrow it down later. The complexity of your research question depends on the type of paper and the instructions you received.

❓ What are typical qualitative research questions?

Typical qualitative research questions begin with words such as "how," "what," "to what extent," etc. They imply that there should be an extended deep answer to the question. In addition, there can be qualitative sub-questions that are narrower than the central question.

❓ What are examples of research questions?

Here are two examples of nursing research questions:

Qualitative research question: What are the consequences of psychological side effects of ADHD medication for children?

Quantitative research question: How do the psychological side effects of ADHD medication for children influence academic performance?

• Qualitative research questions; Scientific Inquiry in Social Work
• How to write Qualitative Research Questions and Questionnaires
• Strategies for Selecting a Research Topic - ResearchGate
• Qualitative or Quantitative Research? | MQHRG - McGill University

• Call to +1 844 889-9952

Scientific Research Question Generator

Feeling stuck trying to make a fresh and creative research question? Try our free research question generator! Choose a suitable question from a list of suggestions or build your own.

• Question Maker
• Question Finder

Make a scientific research question with this tool in 3 simple steps:

Please try again with some different keywords.

• 🧪 What Is This Tool?

‼️ Why Are Research Questions Important?

• 📃 How to Create a Science Paper?
• 🔗 References

🧪 Scientific Research Question Generator: What Is It?

Welcome to the page of our scientific research question generator! Right about now, you’re probably wondering – what is this tool, and how does it work? We present you with two options – a generator and a builder. You can read more about them below.

Scientific Question Generator

Deciding to use a question generator is a great alternative to save time and get what you want. You won’t have to suffer for hours looking for a fresh and creative idea! Once you customize the generator to your requirements, you’ll get incredible results.

What is good about this option? Simply put, you’ll only need to follow a few basic steps to create a research question. First, enter the keywords for your future work. You can also select a research area to optimize the generator’s search. Run a search for results and choose a question option from the many suggested ideas! You can refresh your search until you find the research question that fits and inspires you the most.

Research Question Builder

This tool has another feature that may come in handy – a generator of individual research questions from scratch. You don’t need to come up with your own options and guess how to write a well-written idea. It is a valuable function that will save time and produce more creative outcomes. To generate it, you’ll have to specify more qualifying study details.

As the first step, decide your study group and the factor that affects it . Next, try to formulate a measurable outcome of your work. You can add another study group to expand the generator’s capabilities. And finally, specify the time frame of the study. As a result, you have a ready-made individualized research question.

A research question is a helpful tool both for students and researchers. Sound and well-constructed questions are the ones that can shape the structure of your study. They should be grounded in consciously chosen data, instead of random variables. We can use these important questions not only for academic objectives but also in other life situations. For example, by studying the research questions of a potential employer, you can understand the suitability of the company and this kind of job for you.

A well-worded question will be easier for you to answer. You can also use it to outline your research and identify possible problems. That approach will reduce the time it takes to prepare the design of your study. To create a good research question , you need to:

• Choose an area of interest.
• Focus on a specific topic.
• Compose smaller support questions .
• Select the type of data collection and review the applicable literature.
• Identify your target audience.

📃 How to Create a Good Science Paper?

Scientific research papers are similar to the standard essays you are used to writing in school and college. But they have their specificities that you should be aware of. In this section, we have broken down the structure of a typical science paper and explained what goes into each part.

Your title should be specific and concise. It should also describe the subject and be comprehensive. However, it should be clear enough to be understood by a broader target audience, not just narrowly focused specialists.

The abstract is often a necessary component of academic work. The principal aim is to allow the reader a quick look at the scientific material and decide whether they are interested. However, this part shouldn’t be as technical as the main study, so as not to distract them. The abstract consists of general objectives, methods, results, and conclusions, and is approximately 150 to 250 words long. Note that you shouldn’t include citations, notations, and abbreviations.

Introduction

You should write an introduction describing the statement of your problem, and why it’s relevant and worthwhile. A few paragraphs will be enough. You can mention the main sources you have been working with to keep your audience involved. Also, remember to provide the necessary context and background information for your research. You can finish the introduction by explaining the essence of your research question and the value of your answer.

Methods & Materials

In this section of the paper, you should provide the methods and materials you have used for your study. It’s necessary to make your results replicable, and use qualitative or quantitative research methods (or a mix of both). You can use tables, diagrams, and charts to visually represent this information. You shouldn’t disclose your work findings, but you can include preview conclusions for reference.

At this point, we present the final study results, outlining the essential conclusions. Remember, there is no need to discuss the findings or cause-and-effect relationships. Avoid including subtotal results you have received and don’t affect the bottom line. Also, avoid manipulating your audience or exaggerating your achievements, as your results should be testable.

Provide the most meaningful results for discussion . Describe how these results relate to your question and how they are consistent with the results of other researchers. Indicate if the results coincided with your expectations and how you can interpret them. Also, mention if your findings raise issues and how they impact the scope of the study. You may finish up with the relevance of your conclusions.

When you give data in tables or charts, be sure to include a header describing the information in them. Don’t use tables or charts if they are irrelevant. Also, don’t insert them if you need to display data that can fit into a couple of sentences. Make sure to annotate all the visual data you end up using and mention them in the list of figures in the appendix.

Every scientific research paper must have a list of references at the end. This is to avoid plagiarism and to support the validity of your study. Remember to use notations as you go along and indicate them in the text. Then, you must list all the literature used in alphabetical order at the end of the paper. Double-check the citation style of your institution before making this list.

We hope you found our tool helpful in your work! Be sure to check out the FAQ section below if you still have any questions.

❓ Scientific Question Generator – FAQ

❓ how do you develop a scientific question.

Formulate the question in such a way that you can study it. It should be clear, understandable, and brief. After reading your research question, the reader should understand what your paper will be about. Therefore, it should have an objective , relevance, and meaning.

❓ What are good examples of a science research question?

“What are the legal aspects affecting the decrease in people who drive under the influence of alcohol in the USA?” — This question focuses on a defined topic and reviews the effectiveness of existing legislation.  

“How can universities improve the environment for students to become more LGBT-inclusive?” — This question focuses on one specific issue and addresses a narrowly targeted area.

❓ What are the 3 qualities of a good scientific question?

A good question should be feasible in the context of the research accessible to the field of study, ethical, sufficient methods, and materials. It should be interesting, engaging, and intriguing to the target audience. Finally, it should also be relevant and provide new ideas to the chosen field for future research.

Updated: May 17th, 2024

📎 References

• Scientific Writing Made Easy: A Step-by-Step Guide to Undergraduate Writing in the Biological Sciences – Sheela P. Turbek, Taylor M. Chock, Kyle Donahue, Caroline A. Havrilla, Angela M. Oliverio, Stephanie K. Polutchko, Lauren G. Shoemaker & Lara Vimercati, Ecological Society of America
• Writing the Scientific Paper – Emily Wortman-Wunder & Kate Kiefer, Colorado State University
• Organizing Your Social Sciences Research Paper, University of Southern California
• Your research question – Imperial College London
• Developing research questions – Monash University

Research Question Maker

Please try again with some different keywords or subjects.

Looking for a research question maker to get a ready research question or build one from scratch?

Search no more!

This 2-in-1 online research question making tool can do both in seconds.

Try our it and break free from the stressful experience. The tool is user-friendly, and you can easily access it online for free.

• ️🤔 How to Use the Tool?
• ️🕵🏽 What Is a Research Question?
• ️🔢 Research Question Formula
• ️🔎 Research Question Types
• ️✅ Research Question Checklist
• ️👀 Examples
• ️🔗 References

🤔 Research Question Maker: How to Use the 2-in-1 Tool?

Getting a ready research question.

You don't have to stress over our research question generator because you get impressive results within a few seconds.

Get your research question by following the steps below:

• Enter the keywords related to the research question you are interested in exploring.
• Choose your study area if necessary.
• Run the search and wait for the results.
• Look at the many ideas that the question maker will propose.

You can refresh the search button until you find the question that suits your research paper.

Building a Tailor-made Research Question

Another option of this 2-in-1 tool enables you to build a tailor-made research question from scratch. To get one quickly, perform the following steps:

🕵🏽 What Is a Research Question?

A research question is important in guiding your research paper, essay, or thesis. It offers the direction of your research and clarifies what you want to focus on.

Good research questions require you to synthesize facts from several sources and interpret them to get an answer.

It is essential to understand the features of a good research question before you start the formulation process.

Your question should be:

• Focused. It should focus on one research issue.
• Specific. The question should contain narrowed ideas .
• Researchable. You should get answers from qualitative and quantitative research sources .
• Feasible. It should be workable within the practical limitations
• Original. The question should be unique and authentic.
• Relevant. It needs to be based on your subject discipline.
• Complex. It should offer adequate scope for analysis and discussion.

Research papers or essays require one research question, as a rule. However, extensive projects like dissertations and theses can have several research questions focusing on the main research issue.

The thesis statement is the response you develop; it sets the direction of your arguments. It should be relevant to the research question.

Thus, you can also use an online thesis maker to ensure it aligns with your formulated questions.

🔢 Research Question Formula

In research writing , you must begin with a topic of interest. Analyzing the original title, you have chosen will give you a good and well-defined research question.

There is an effective formula you can use when formulating your research question.

Topic + Concept = Research question

The topic should be specific with a strong focus on a subject matter, while the concept surrounding it should be from a broad field.

For example:

Your topic could be social media, nursing, standardized tests, cybersecurity, etc. Conversely, concepts can be the risks and benefits of your topic, the recent trends, challenges faced by the industry, etc.

Let us explore the formula and create a few research questions.

• Standardized tests (topic) + recent trends (concept) = How have standardized tests impacted the education sector? (research question)
• Cybersecurity (topic) + effect (concept) = How has cybersecurity affected the evolution of technology? (research question)

Therefore, ensure your research question is neither too broad nor too narrow. Broad topics and concepts might overwhelm you with numerous sources. On the other hand, narrow questions will limit you when exploring the project's scope.

🔎 Research Question Types

When formulating your research question, choose from 3 fundamental types that your academic paper can focus on.

Descriptive Research Question

When your investigation intends to disclose existing patterns within the research subject, you should use this type.

A descriptive question urges you to collect measurable information about the attributes of subjects with certain views. It could be a number, occurrence, or amount that describes a research problem.

Here are some examples:

• What is the percentage of people with fitness apps in 2022?
• What is the average debt load of an American?
• How often do students use online writing services in the UK?

Relational Research Question

This type focuses on comparing two or more entities in a research investigation. After picking your variables, you must choose a comparison parameter and provide its detailed discussion.

Some examples are as follows:

• What is the difference between men and women's salaries in IT?
• What is the correlation between alcohol and depression?
• Is there a relationship between a vegan diet and the low-income bracket?

Causal Research Question

This is a cause-and-effect type of research question. It seeks to prove how one variable affects another one.

Great examples are:

• How does advertising impact consumer behavior?
• Do public opinion polls alter voter inclinations?
• How does employee training affect performance in the employment market?

✅ How to Make a Research Question Stronger? The Checklist

Developing questions seems like a simple task for students. But it can be quite challenging if you want to create an effective research question. The latter can make or break your paper, so you should focus on strengthening and refining it.

How do you make your research question strong? The criteria below will show whether you've already arrived at a workable question.

👀 Research Question Examples

• What does a change-ready organization look like?
• Wearable medical devices: how will they transform healthcare?
• What effect did the World War II wartime experience have on African americans?
• Biodiversity on the Earth: why is it crucial for the environment?
• What makes William Shakespeare relevant in the modern day?
• How did the Civil War affect the distribution of wealth in the United States?
• What is love?
• Why should businesses embrace remote work?
• What impact has feminism had in the study of women and crime?
• How to construct a mixed methods research design?
• What is a halogenated hydrocarbon?

Thank you for reading this article! If you need to formulate a research title, try using our title-generating tool .

❓ Research Question Maker Tool FAQ

❓ what is a good research question.

A great research question is specific and answerable within a workable time frame. It should focus on one topic and be researchable using primary and secondary data. In short, it should have a clear statement of what the researcher is supposed to do to get practical answers.

❓ How to formulate a research question?

To understand how to create a research question, you need to think about how your topic affects a particular population. You should also consider the period of investigation and the location – it could be an organization, country, or commercial industry.

❓ How to write a qualitative research question?

Your questions should reveal research issues and opinions from the respondents in your study. Qualitative questions seek to discover and understand how people make sense of their life experiences and events. The results of qualitative research are analyzed narratively, so don't try to quantify them.

❓ How to find a research question?

If you find it difficult to compose a unique research question, use our question maker tool and get it within a few seconds. Just enter the right keywords about your subject of interest, and the smart algorithms will produce a list of questions that suit your case.

🔗 References

• How to Write a Research Question - GMU Writing Center
• How to Write a Research Question: Steps and Examples
• Narrowing a Topic and Developing a Research Question
• Formulation of Research Question – Stepwise Approach - PMC
• Writing Research Questions: Purpose & Examples - Study.com

• News & Highlights

• Publications and Documents
• Postgraduate Education
• Browse Our Courses
• C/T Research Academy
• K12 Investigator Training
• Harvard Catalyst On-Demand
• Translational Innovator
• SMART IRB Reliance Request
• Biostatistics Consulting
• Regulatory Support
• Pilot Funding
• Informatics Program
• Community Engagement
• Diversity Inclusion
• Research Enrollment and Diversity
• Harvard Catalyst Profiles

Creating a Good Research Question

• Advice & Growth
• Process in Practice

Successful translation of research begins with a strong question. How do you get started? How do good research questions evolve? And where do you find inspiration to generate good questions in the first place?  It’s helpful to understand existing frameworks, guidelines, and standards, as well as hear from researchers who utilize these strategies in their own work.

In the fall and winter of 2020, Naomi Fisher, MD, conducted 10 interviews with clinical and translational researchers at Harvard University and affiliated academic healthcare centers, with the purpose of capturing their experiences developing good research questions. The researchers featured in this project represent various specialties, drawn from every stage of their careers. Below you will find clips from their interviews and additional resources that highlight how to get started, as well as helpful frameworks and factors to consider. Additionally, visit the Advice & Growth section to hear candid advice and explore the Process in Practice section to hear how researchers have applied these recommendations to their published research.

• Naomi Fisher, MD , is associate professor of medicine at Harvard Medical School (HMS), and clinical staff at Brigham and Women’s Hospital (BWH). Fisher is founder and director of Hypertension Services and the Hypertension Specialty Clinic at the BWH, where she is a renowned endocrinologist. She serves as a faculty director for communication-related Boundary-Crossing Skills for Research Careers webinar sessions and the Writing and Communication Center .
• Christopher Gibbons, MD , is associate professor of neurology at HMS, and clinical staff at Beth Israel Deaconess Medical Center (BIDMC) and Joslin Diabetes Center. Gibbons’ research focus is on peripheral and autonomic neuropathies.
• Clare Tempany-Afdhal, MD , is professor of radiology at HMS and the Ferenc Jolesz Chair of Research, Radiology at BWH. Her major areas of research are MR imaging of the pelvis and image- guided therapy.
• David Sykes, MD, PhD , is assistant professor of medicine at Massachusetts General Hospital (MGH), he is also principal investigator at the Sykes Lab at MGH. His special interest area is rare hematologic conditions.
• Elliot Israel, MD , is professor of medicine at HMS, director of the Respiratory Therapy Department, the director of clinical research in the Pulmonary and Critical Care Medical Division and associate physician at BWH. Israel’s research interests include therapeutic interventions to alter asthmatic airway hyperactivity and the role of arachidonic acid metabolites in airway narrowing.
• Jonathan Williams, MD, MMSc , is assistant professor of medicine at HMS, and associate physician at BWH. He focuses on endocrinology, specifically unravelling the intricate relationship between genetics and environment with respect to susceptibility to cardiometabolic disease.
• Junichi Tokuda, PhD , is associate professor of radiology at HMS, and is a research scientist at the Department of Radiology, BWH. Tokuda is particularly interested in technologies to support image-guided “closed-loop” interventions. He also serves as a principal investigator leading several projects funded by the National Institutes of Health and industry.
• Osama Rahma, MD , is assistant professor of medicine at HMS and clinical staff member in medical oncology at Dana-Farber Cancer Institute (DFCI). Rhama is currently a principal investigator at the Center for Immuno-Oncology and Gastroenterology Cancer Center at DFCI. His research focus is on drug development of combinational immune therapeutics.
• Sharmila Dorbala, MD, MPH , is professor of radiology at HMS and clinical staff at BWH in cardiovascular medicine and radiology. She is also the president of the American Society of Nuclear Medicine. Dorbala’s specialty is using nuclear medicine for cardiovascular discoveries.
• Subha Ramani, PhD, MBBS, MMed , is associate professor of medicine at HMS, as well as associate physician in the Division of General Internal Medicine and Primary Care at BWH. Ramani’s scholarly interests focus on innovative approaches to teaching, learning and assessment of clinical trainees, faculty development in teaching, and qualitative research methods in medical education.
• Ursula Kaiser, MD , is professor at HMS and chief of the Division of Endocrinology, Diabetes and Hypertension, and senior physician at BWH. Kaiser’s research focuses on understanding the molecular mechanisms by which pulsatile gonadotropin-releasing hormone regulates the expression of luteinizing hormone and follicle-stimulating hormone genes.

Insights on Creating a Good Research Question

Play Junichi Tokuda video

Play Ursula Kaiser video

Start Successfully: Build the Foundation of a Good Research Question

Start Successfully Resources

Ideation in Device Development: Finding Clinical Need Josh Tolkoff, MS A lecture explaining the critical importance of identifying a compelling clinical need before embarking on a research project. Play Ideation in Device Development video .

Radical Innovation Jeff Karp, PhD This ThinkResearch podcast episode focuses on one researcher’s approach using radical simplicity to break down big problems and questions. Play Radical Innovation .

Using Healthcare Data: How can Researchers Come up with Interesting Questions? Anupam Jena, MD, PhD Another ThinkResearch podcast episode addresses how to discover good research questions by using a backward design approach which involves analyzing big data and allowing the research question to unfold from findings. Play Using Healthcare Data .

Important Factors: Consider Feasibility and Novelty

Refining Your Research Question 

Play video of Clare Tempany-Afdhal

Play Elliott Israel video

Frameworks and Structure: Evaluate Research Questions Using Tools and Techniques

Frameworks and Structure Resources

Designing Clinical Research Hulley et al. A comprehensive and practical guide to clinical research, including the FINER framework for evaluating research questions. Learn more about the book .

Translational Medicine Library Guide Queens University Library An introduction to popular frameworks for research questions, including FINER and PICO. Review translational medicine guide .

Asking a Good T3/T4 Question  Niteesh K. Choudhry, MD, PhD This video explains the PICO framework in practice as participants in a workshop propose research questions that compare interventions. Play Asking a Good T3/T4 Question video

Introduction to Designing & Conducting Mixed Methods Research An online course that provides a deeper dive into mixed methods’ research questions and methodologies. Learn more about the course

Network and Support: Find the Collaborators and Stakeholders to Help Evaluate Research Questions

Network & Support Resource

Bench-to-bedside, Bedside-to-bench Christopher Gibbons, MD In this lecture, Gibbons shares his experience of bringing research from bench to bedside, and from bedside to bench. His talk highlights the formation and evolution of research questions based on clinical need. Play Bench-to-bedside. 

Research Question Generator Online

Are you looking for effective aid in research question formulation? Try our research question generator and get ideas for any project instantly.

• 🤖 How to Use the Tool

❗ Why Is a Research Question Important?

🔖 research question types & examples, 🗺️ how to generate a research question.

• 👀 More Examples
• 🔍 References

🤖 How to Use a Research Question Generator?

Struggling to develop a good research question for your college essay , proposal , or dissertation ? Don't waste time anymore, as our research question generator is available online for free.

Our tool is designed to provide original questions to suit any subject discipline.

Generate your questions in a few easy steps as shown below:

• Add your research group and the influencing factor.
• Indicate your dependent variable (the thing you’re planning to measure).
• Add the optional parameters (the second research group and the time frame).
• Look at the examples if necessary.

Once you get the initial results, you can still refine the questions to get relevant and practical research questions for your project.

The main importance of formulating a research question is to break down a broad topic and narrow it to a specific field of investigation . It helps you derive a practical knowledge of the topic of interest. The research question also acts as a guiding structure for the entire investigation from paragraph to paragraph. Besides, you can define research issues and spot gaps in the study.

The research questions disclose the boundaries and limitations of your research, ensuring it is consistent and relevant. Ultimately, these questions will directly affect the research methods you will use to collect and analyze data. They also affect the process of generating a thesis statement . With a checker proposal, you can also polish your research question to ensure it aligns with the research purpose.

The research writing process covers different types of questions, depending on the depth of study and subject matter. It is important to know the kind of research you want to do; it will help you in the formulation of an effective research question. You can select quantitative, qualitative, or mixed methods studies to develop your questions.

Let us explore some of these question types in detail to help you choose a workable option for your project:

Quantitative Research Questions

Quantitative questions are specific and objective, providing detailed information about a particular research topic . The data you collect from this research type is quantifiable and can be studied using figures.

These questions also delineate a relationship between the research design and the research question.

Quantitative questions focus on issues like:

• "How often"
• "How intense"
• "Is there a statistical relationship"

They illustrate the response with numbers.

In addition, quantitative questions help you to explore existing patterns in data from a specific location or context. The collected information allows researchers to make logical and data-driven conclusions.

This type of research question can be classified further into 3 categories.

Descriptive Research Questions

Such questions seek to describe a quantifiable problem and investigate the numbers, rates, or intensity of the issue. They are usually used to write descriptive papers .

Comparative Research Questions

As the name suggests, comparative questions intend to compare and contrast two or more issues in a research project. These questions are used in comparative papers . To formulate such a question, identify two or more variables, choose a standard for comparison, and present an in-depth discussion.

Let's look at a few examples.

Relationship-based Research Questions

Relationship-based questions reveal and identify a connection between two or more research variables . Such questions entail a dependent variable, an independent variable, and a socio-demographic of the population you are interested in studying.

Qualitative Research Questions

Qualitative research questions are open-ended and aim to explore or explain respondents' subjective meanings and experiences . You can't measure the data you collect from a qualitative research question in figures, as it's mostly narrative. Some of the common types include those described below.

Exploratory Research Questions

These questions investigate a particular research topic without any assumptions.

Explanatory Research Questions

These questions examine the reasons and find connections between existing entities.

Mixed Methods Studies

When you combine quantitative and qualitative research questions, you will get a mixed-method research study . It answers your research question more comprehensively since it combines the advantages of both research methods in a pragmatic study .

This mixed study can focus on quantitative data (score comparison with attitude ranking) and qualitative insights from student interviews about attitudes.

We have outlined a few steps to generate exceptional questions for students who don't know how to write them effectively.

👀 More Research Question Examples

• Why do minorities delay going to the doctor?
• What makes humans mortal genetically?
• Why and how did the US get involved in the Korean War?
• The virus COVID-19: what went wrong?
• What is cancel culture, and can it go too far?
• How do human infants acquire a language?
• Eastern vs. Western religions: what’s the difference?
• Why is capitalism better than socialism?
• What do Hamlet and Oedipus have in common?
• How does language influence our world?
• Competence for nurses: why is it important?
• COVID-19 pandemic: what we can learn from the past?

❓ Research Question Generator FAQ

❓ how to form a research question.

You should select an interesting topic related to the subject you are studying. Carry out preliminary research with our research question generator online and pick the question from the list of offered suggestions. Refine the question until you are satisfied with the result.

❓ What makes a good research question?

An effective research question should focus on a single issue and clearly state the research direction you will take. The topic should neither be too broad nor too narrow – just enough to keep you focused on the main scope of the study. Also, it should be answerable with a comprehensive analysis.

❓ How to find the research question in an article?

In an academic article, the research question is usually placed at the end of the introduction, right before the literature review. At times, it may be included in the methods section – after the review of academic research.

❓ How to write a quantitative research question?

Identify what claim you want to make in your research purpose. Choose a dependent variable, an independent variable, and a target population, and formulate the assumed relationship between the variables for that respondent group. Ensure the data you collect is measured within a specific context.

🔗 References

• Types of Research Questions With Examples
• Developing research questions - Library - Monash University
• Research Question - Research Guide - LibGuides
• How To Write a Research Question: Steps and Examples
• How to Write a Research Question - GMU Writing Center

We use cookies and similar technologies to improve your website experience and help us understand how you use our website. By continuing to use this website, you consent to the usage of cookies. Learn more about our Privacy Statement and Cookie Policy .

• Our Mission
• Code of Conduct
• The Consultants
• Hours and Locations
• Apply to Become a Consultant
• Make an Appointment
• Face-to-Face Appointments
• Zoom Appointments
• Written Feedback Appointments
• Support for Writers with Disabilities
• Policies and Restrictions
• Upcoming Workshops
• Class Workshops
• Meet the Consultants
• Writing Guides and Tools
• Schedule an appointment! Login or Register
• Graduate Students
• ESOL Students

How to Write a Research Question

What is a research question? A research question is the question around which you center your research. It should be:

• clear : it provides enough specifics that one’s audience can easily understand its purpose without needing additional explanation.
• focused : it is narrow enough that it can be answered thoroughly in the space the writing task allows.
• concise : it is expressed in the fewest possible words.
• complex : it is not answerable with a simple “yes” or “no,” but rather requires synthesis and analysis of ideas and sources prior to composition of an answer.
• arguable : its potential answers are open to debate rather than accepted facts.

You should ask a question about an issue that you are genuinely curious and/or passionate about.

The question you ask should be developed for the discipline you are studying. A question appropriate for Biology, for instance, is different from an appropriate one in Political Science or Sociology. If you are developing your question for a course other than first-year composition, you may want to discuss your ideas for a research question with your professor.

Why is a research question essential to the research process? Research questions help writers focus their research by providing a path through the research and writing process. The specificity of a well-developed research question helps writers avoid the “all-about” paper and work toward supporting a specific, arguable thesis.

Steps to developing a research question:

• Choose an interesting general topic. Most professional researchers focus on topics they are genuinely interested in studying. Writers should choose a broad topic about which they genuinely would like to know more. An example of a general topic might be “Slavery in the American South” or “Films of the 1930s.”
• Do some preliminary research on your general topic. Do a few quick searches in current periodicals and journals on your topic to see what’s already been done and to help you narrow your focus. What issues are scholars and researchers discussing, when it comes to your topic? What questions occur to you as you read these articles?
• Consider your audience. For most college papers, your audience will be academic, but always keep your audience in mind when narrowing your topic and developing your question. Would that particular audience be interested in the question you are developing?
• Start asking questions. Taking into consideration all of the above, start asking yourself open-ended “how” and “why” questions about your general topic. For example, “Why were slave narratives effective tools in working toward the abolishment of slavery?” or “How did the films of the 1930s reflect or respond to the conditions of the Great Depression?”
• Is your research question clear? With so much research available on any given topic, research questions must be as clear as possible in order to be effective in helping the writer direct his or her research.
• Is your research question focused? Research questions must be specific enough to be well covered in the space available.
• Is your research question complex? Research questions should not be answerable with a simple “yes” or “no” or by easily-found facts.  They should, instead, require both research and analysis on the part of the writer. They often begin with “How” or “Why.”
• Begin your research . After you’ve come up with a question, think about the possible paths your research could take. What sources should you consult as you seek answers to your question? What research process will ensure that you find a variety of perspectives and responses to your question?

Sample Research Questions

Unclear: How should social networking sites address the harm they cause? Clear: What action should social networking sites like MySpace and Facebook take to protect users’ personal information and privacy? The unclear version of this question doesn’t specify which social networking sites or suggest what kind of harm the sites might be causing. It also assumes that this “harm” is proven and/or accepted. The clearer version specifies sites (MySpace and Facebook), the type of potential harm (privacy issues), and who may be experiencing that harm (users). A strong research question should never leave room for ambiguity or interpretation. Unfocused: What is the effect on the environment from global warming? Focused: What is the most significant effect of glacial melting on the lives of penguins in Antarctica?

The unfocused research question is so broad that it couldn’t be adequately answered in a book-length piece, let alone a standard college-level paper. The focused version narrows down to a specific effect of global warming (glacial melting), a specific place (Antarctica), and a specific animal that is affected (penguins). It also requires the writer to take a stance on which effect has the greatest impact on the affected animal. When in doubt, make a research question as narrow and focused as possible.

Too simple: How are doctors addressing diabetes in the U.S.? Appropriately Complex:   What main environmental, behavioral, and genetic factors predict whether Americans will develop diabetes, and how can these commonalities be used to aid the medical community in prevention of the disease?

The simple version of this question can be looked up online and answered in a few factual sentences; it leaves no room for analysis. The more complex version is written in two parts; it is thought provoking and requires both significant investigation and evaluation from the writer. As a general rule of thumb, if a quick Google search can answer a research question, it’s likely not very effective.

Last updated 8/8/2018

The Writing Center

4400 University Drive, 2G8 Fairfax, VA 22030

• Johnson Center, Room 227E
• +1-703-993-1200
• [email protected]

Quick Links

• Register with us

© Copyright 2024 George Mason University . All Rights Reserved. Privacy Statement | Accessibility

Library buildings are open for UniSA staff and students via UniSA ID swipe cards.   Please contact us on Ask the Library for any assistance. Find out about other changes to Library services .

Develop your research question

• Search for your assignment
• Find books and journal articles
• Find government and organisation information
• Properties and calculations
• Drug information
• Assignment Help

STEP 1: Understand your research objective

Before you start developing your research question, think about your research objectives:

• What are you trying to do? (compare, analyse)
• What do you need to know about the topic?
• What type of research are you doing?
• What types of information/studies do you need? (e.g. randomised controlled trial, case study, guideline, protocol?)
• Does the information need to be current?

Watch the following video (6:26) to get you started:

Key points from the video

• All good academic research starts with a research question.
• A research question is an actual question you want to answer about a particular topic.
• Developing a question helps you focus on an aspect of your topic, which will streamline your research and writing.
• Pick a topic you are interested in.
• Narrow the topic to a particular aspect.
• Brainstorm some questions around your topic aspect.
• Select a question to work with.
• Focus the question by making it more specific. Make sure your question clearly states who, what, when, where, and why.
• A good research question focuses on one issue only and requires analysis.
• Your search for information should be directed by your research question.
• Your thesis or hypothesis should be a direct answer to your research question, summarised into one sentence.

STEP 2: Search before you research

The benefits of doing a background search :

• You can gather more background knowledge on a subject
• explore different aspects of your topic
• identify additional keywords and terminology

STEP 3: Choose a topic

The resources linked below are a good place to start: 

• UpToDate It covers thousands of clinical topics grouped into specialties with links to articles, drugs and drug interaction databases, medical calculators and guidelines.
• An@tomedia This online anatomy resource features images, videos, and slides together with interactive, educational text and quiz questions.
• Anatomy.tv Find 3D anatomical images; functional anatomy animations and videos, and MRI, anatomy, and clinical slides. Test your knowledge through interactive activities and quizzes.

STEP 4: Brainstorm your questions

Now you have explored different aspects of your topic, you may construct more focused questions (you can create a few questions and pick one later).

Learn more: 

• Clear and present questions: formulating questions for evidence based practice (Booth 2006) This article provides an overview of thinking in relation to the theory and practice of formulating answerable research questions.

STEP 5: Pick a question and focus

Once you have a few questions to choose from, pick one and refine it even further.

Are you required to use "PICO"?

• PICO worksheet
• Other frameworks

The PICO framework (or other variations) can be useful for developing an answerable clinical question. 

The example question used in this guide is a PICO question:   How does speech therapy compare to cognitive behavioural therapy in improving speech fluency in adolescents?

Use the interactive PICO worksheet to get started with your question, or you can download the worksheet document.

• Building your question with PICO

Here are some different frameworks you may want to use:

There are a number of PICO variations which can be used for different types of questions, such as qualitative, and background and foreground questions. Visit the Evidence-Based Practice (EBP) Guide to learn more:

• Evidence Based Practice guide
• << Previous: Plan your search
• Next: Search for your assignment >>
• Last Updated: Apr 18, 2024 10:51 AM
• URL: https://guides.library.unisa.edu.au/Chemistry

The text within this Guide is licensed CC BY 4.0 . Image licenses can be found within the image attributions document on the last page of the Guide. Ask the Library for information about reuse rights for other content within this Guide.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 2: Getting Started in Research

Generating Good Research Questions

Learning Objectives

• Describe some common sources of research ideas and generate research ideas using those sources.
• Describe some techniques for turning research ideas into empirical research questions and use those techniques to generate questions.
• Explain what makes a research question interesting and evaluate research questions in terms of their interestingness.

Good research must begin with a good research question. Yet coming up with good research questions is something that novice researchers often find difficult and stressful. One reason is that this is a creative process that can appear mysterious—even magical—with experienced researchers seeming to pull interesting research questions out of thin air. However, psychological research on creativity has shown that it is neither as mysterious nor as magical as it appears. It is largely the product of ordinary thinking strategies and persistence (Weisberg, 1993) [1] . This section covers some fairly simple strategies for finding general research ideas, turning those ideas into empirically testable research questions, and finally evaluating those questions in terms of how interesting they are and how feasible they would be to answer.

Finding Inspiration

Research questions often begin as more general research ideas—usually focusing on some behaviour or psychological characteristic: talkativeness, learning, depression, bungee jumping, and so on. Before looking at how to turn such ideas into empirically testable research questions, it is worth looking at where such ideas come from in the first place. Three of the most common sources of inspiration are informal observations, practical problems, and previous research.

Informal observations include direct observations of our own and others’ behaviour as well as secondhand observations from nonscientific sources such as newspapers, books, blogs, and so on. For example, you might notice that you always seem to be in the slowest moving line at the grocery store. Could it be that most people think the same thing? Or you might read in a local newspaper about people donating money and food to a local family whose house has burned down and begin to wonder about who makes such donations and why. Some of the most famous research in psychology has been inspired by informal observations. Stanley Milgram’s famous research on obedience to authority, for example, was inspired in part by journalistic reports of the trials of accused Nazi war criminals—many of whom claimed that they were only obeying orders. This led him to wonder about the extent to which ordinary people will commit immoral acts simply because they are ordered to do so by an authority figure (Milgram, 1963) [2] .

Practical problems can also inspire research ideas, leading directly to applied research in such domains as law, health, education, and sports. Does taking lecture notes by hand improve students’ exam performance? How effective is psychotherapy for depression compared to drug therapy? To what extent do cell phones impair people’s driving ability? How can we teach children to read more efficiently? What is the best mental preparation for running a marathon?

Probably the most common inspiration for new research ideas, however, is previous research. Recall that science is a kind of large-scale collaboration in which many different researchers read and evaluate each other’s work and conduct new studies to build on it. Of course, experienced researchers are familiar with previous research in their area of expertise and probably have a long list of ideas. This suggests that novice researchers can find inspiration by consulting with a more experienced researcher (e.g., students can consult a faculty member). But they can also find inspiration by picking up a copy of almost any professional journal and reading the titles and abstracts. In one typical issue of  Psychological Science , for example, you can find articles on the perception of shapes, anti-Semitism, police lineups, the meaning of death, second-language learning, people who seek negative emotional experiences, and many other topics. If you can narrow your interests down to a particular topic (e.g., memory) or domain (e.g., health care), you can also look through more specific journals, such as  Memory & Cognition  or  Health Psychology .

Generating Empirically Testable Research Questions

Once you have a research idea, you need to use it to generate one or more empirically testable research questions, that is, questions expressed in terms of a single variable or relationship between variables. One way to do this is to look closely at the discussion section in a recent research article on the topic. This is the last major section of the article, in which the researchers summarize their results, interpret them in the context of past research, and suggest directions for future research. These suggestions often take the form of specific research questions, which you can then try to answer with additional research. This can be a good strategy because it is likely that the suggested questions have already been identified as interesting and important by experienced researchers.

But you may also want to generate your own research questions. How can you do this? First, if you have a particular behaviour or psychological characteristic in mind, you can simply conceptualize it as a variable and ask how frequent or intense it is. How many words on average do people speak per day? How accurate are our memories of traumatic events? What percentage of people have sought professional help for depression? If the question has never been studied scientifically—which is something that you will learn in your literature review—then it might be interesting and worth pursuing.

If scientific research has already answered the question of how frequent or intense the behaviour or characteristic is, then you should consider turning it into a question about a statistical relationship between that behaviour or characteristic and some other variable. One way to do this is to ask yourself the following series of more general questions and write down all the answers you can think of.

• What are some possible causes of the behaviour or characteristic?
• What are some possible effects of the behaviour or characteristic?
• What types of people might exhibit more or less of the behaviour or characteristic?
• What types of situations might elicit more or less of the behaviour or characteristic?

In general, each answer you write down can be conceptualized as a second variable, suggesting a question about a statistical relationship. If you were interested in talkativeness, for example, it might occur to you that a possible cause of this psychological characteristic is family size. Is there a statistical relationship between family size and talkativeness? Or it might occur to you that people seem to be more talkative in same-sex groups than mixed-sex groups. Is there a difference in the average level of talkativeness of people in same-sex groups and people in mixed-sex groups? This approach should allow you to generate many different empirically testable questions about almost any behaviour or psychological characteristic.

If through this process you generate a question that has never been studied scientifically—which again is something that you will learn in your literature review—then it might be interesting and worth pursuing. But what if you find that it has been studied scientifically? Although novice researchers often want to give up and move on to a new question at this point, this is not necessarily a good strategy. For one thing, the fact that the question has been studied scientifically and the research published suggests that it is of interest to the scientific community. For another, the question can almost certainly be refined so that its answer will still contribute something new to the research literature. Again, asking yourself a series of more general questions about the statistical relationship is a good strategy.

• Are there other ways to operationally define the variables?
• Are there types of people for whom the statistical relationship might be stronger or weaker?
• Are there situations in which the statistical relationship might be stronger or weaker—including situations with practical importance?

For example, research has shown that women and men speak about the same number of words per day—but this was when talkativeness was measured in terms of the number of words spoken per day among university students in the United States and Mexico. We can still ask whether other ways of measuring talkativeness—perhaps the number of different people spoken to each day—produce the same result. Or we can ask whether studying elderly people or people from other cultures produces the same result. Again, this approach should help you generate many different research questions about almost any statistical relationship.

Evaluating Research Questions

Researchers usually generate many more research questions than they ever attempt to answer. This means they must have some way of evaluating the research questions they generate so that they can choose which ones to pursue. In this section, we consider two criteria for evaluating research questions: the interestingness of the question and the feasibility of answering it.

Interestingness

How often do people tie their shoes? Do people feel pain when you punch them in the jaw? Are women more likely to wear makeup than men? Do people prefer vanilla or chocolate ice cream? Although it would be a fairly simple matter to design a study and collect data to answer these questions, you probably would not want to because they are not interesting. We are not talking here about whether a research question is interesting to us personally but whether it is interesting to people more generally and, especially, to the scientific community. But what makes a research question interesting in this sense? Here we look at three factors that affect the  interestingness  of a research question: the answer is in doubt, the answer fills a gap in the research literature, and the answer has important practical implications.

First, a research question is interesting to the extent that its answer is in doubt. Obviously, questions that have been answered by scientific research are no longer interesting as the subject of new empirical research. But the fact that a question has not been answered by scientific research does not necessarily make it interesting. There has to be some reasonable chance that the answer to the question will be something that we did not already know. But how can you assess this before actually collecting data? One approach is to try to think of reasons to expect different answers to the question—especially ones that seem to conflict with common sense. If you can think of reasons to expect at least two different answers, then the question might be interesting. If you can think of reasons to expect only one answer, then it probably is not. The question of whether women are more talkative than men is interesting because there are reasons to expect both answers. The existence of the stereotype itself suggests the answer could be yes, but the fact that women’s and men’s verbal abilities are fairly similar suggests the answer could be no. The question of whether people feel pain when you punch them in the jaw is not interesting because there is absolutely no reason to think that the answer could be anything other than a resounding yes.

A second important factor to consider when deciding if a research question is interesting is whether answering it will fill a gap in the research literature. Again, this means in part that the question has not already been answered by scientific research. But it also means that the question is in some sense a natural one for people who are familiar with the research literature. For example, the question of whether taking lecture notes by hand can help improve students’ exam performance would be likely to occur to anyone who was familiar with research on notetaking and the ineffectiveness of shallow processing on learning.

A final factor to consider when deciding whether a research question is interesting is whether its answer has important practical implications. Again, the question of whether taking notes by hand improves learning has important implications for education, including classroom policies concerning technology use. The question of whether cell phone use impairs driving is interesting because it is relevant to the personal safety of everyone who travels by car and to the debate over whether cell phone use should be restricted by law.

Feasibility

A second important criterion for evaluating research questions is the feasibility  of successfully answering them. There are many factors that affect feasibility, including time, money, equipment and materials, technical knowledge and skill, and access to research participants. Clearly, researchers need to take these factors into account so that they do not waste time and effort pursuing research that they cannot complete successfully.

Looking through a sample of professional journals in psychology will reveal many studies that are complicated and difficult to carry out. These include longitudinal designs in which participants are tracked over many years, neuroimaging studies in which participants’ brain activity is measured while they carry out various mental tasks, and complex nonexperimental studies involving several variables and complicated statistical analyses. Keep in mind, though, that such research tends to be carried out by teams of highly trained researchers whose work is often supported in part by government and private grants. Keep in mind also that research does not have to be complicated or difficult to produce interesting and important results. Looking through a sample of professional journals will also reveal studies that are relatively simple and easy to carry out—perhaps involving a convenience sample of university students and a paper-and-pencil task.

A final point here is that it is generally good practice to use methods that have already been used successfully by other researchers. For example, if you want to manipulate people’s moods to make some of them happy, it would be a good idea to use one of the many approaches that have been used successfully by other researchers (e.g., paying them a compliment). This is good not only for the sake of feasibility—the approach is “tried and true”—but also because it provides greater continuity with previous research. This makes it easier to compare your results with those of other researchers and to understand the implications of their research for yours, and vice versa.

Key Takeaways

• Research ideas can come from a variety of sources, including informal observations, practical problems, and previous research.
• Research questions expressed in terms of variables and relationships between variables can be suggested by other researchers or generated by asking a series of more general questions about the behaviour or psychological characteristic of interest.
• It is important to evaluate how interesting a research question is before designing a study and collecting data to answer it. Factors that affect interestingness are the extent to which the answer is in doubt, whether it fills a gap in the research literature, and whether it has important practical implications.
• It is also important to evaluate how feasible a research question will be to answer. Factors that affect feasibility include time, money, technical knowledge and skill, and access to special equipment and research participants.
• Practice: Generate five research ideas based on each of the following: informal observations, practical problems, and topics discussed in recent issues of professional journals.
• Practice: Generate five empirical research questions about each of the following behaviours or psychological characteristics: long-distance running, getting tattooed, social anxiety, bullying, and memory for early childhood events.
• Practice: Evaluate each of the research questions you generated in Exercise 2 in terms of its interestingness based on the criteria discussed in this section.
• Practice: Find an issue of a journal that publishes short empirical research reports (e.g.,  Psychological Science ,  Psychonomic Bulletin and Review , Personality and Social Psychology Bulletin ). Pick three studies, and rate each one in terms of how feasible it would be for you to replicate it with the resources available to you right now. Use the following rating scale: (1) You could replicate it essentially as reported. (2) You could replicate it with some simplifications. (3) You could not replicate it. Explain each rating.

Video Attributions

• “ How to Develop a Good Research Topic ” by KStateLibraries . CC BY (Attribution)
• Weisberg, R. W. (1993). Creativity: Beyond the myth of genius . New York, NY: Freeman. ↵
• Milgram, S. (1963). Behavioural study of obedience. Journal of Abnormal and Social Psychology, 67 , 371–378. ↵

The level a research question is interesting to the scientific community and people in general.

the state or ability of being easily or conveniently completed.

Research Methods in Psychology - 2nd Canadian Edition Copyright © 2015 by Paul C. Price, Rajiv Jhangiani, & I-Chant A. Chiang is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

• Jul 7, 2020

How to Write a Science Research Question

Humans are a very curious species. We are always asking questions. But the way we formulate a question is very important when we think about science and research. Here we’ll lay out how to form a science research question and the concepts needed to formulate a good research question. Luckily, we’ve got some handy visuals to help you along.

In order to inquire about the world, produce new information, and solve a mystery of about the natural world, we always use the scientific process to inform research questions. So, we need to keep in mind the steps of the scientific process :

Observation

Data to be obtained

Ways to analyze data

Conclusions to obtain from the question

First, clearly define your population and your variables.

Now, what is a population ? Defined in ecologic terms, a population are all the individuals of one species in a given area (e.g. population of deer, leatherback turtles, spruce trees, mushrooms, etc.).

Now, what is a variable ? A variable is any factor, trait, or condition that can exist in differing amounts or types (e.g. length, quantity, temperature, speed, mass, distance, depth, etc.).

So, using different combinations of these two components, we can create three different types of research questions: descriptive, comparative, and correlative. These three types also match three of the modern research methodologies. 

Descriptive field investigations involve describing and/or quantifying parts of a natural system. Includes generally 1 population and one distinctive variable (figure 1). Examples of descriptive research questions:

How many pine trees are in the Mammoth Hot Springs area?

What is the wolf pack’s distribution range?

How frequently do humpback whales breed?    

Comparative field investigations involve collecting data on different populations/organisms, or under different conditions (e.g., times of year, locations), to make a comparison. Includes two or more populations and one distinctive variable (figure 2). Examples of comparative research questions:

Is there a difference in body length between male and female tortoises?

Is there a difference in diversity of fungi that live in the forest compared with non-forested areas?  

Correlative field investigations involve measuring or observing two variables and searching for a relationship between them for a distinctive population (figure 3). Examples of correlative research questions:

What is the relationship between length of the tail and age in humpback whales?

How does a spider’s reproduction rate change with a change in season?

To practice how to write a research question, we suggest the following steps:

Find a nice place where you can be alone and connected with nature. Bring nothing else but a journal and a pencil. Take a few moments to breath and observe everything that surrounds you. Use all of your senses to obtain information from your surroundings: smell the flowers around you, feel the leaves, hear the birds, and recognize all the life.

Choose a population that is around you and that interests you (flowers, trees, insects, rocks), and think about what would you like to know about that population. Write down what you want to study from that population (your variable). It is easier to choose the population first and the variables second. Think about a feasible and simple measurement. One easy measurement is counting, since it doesn’t require an instrument.

Write down your question using your population and variable. Remember to write a question that is going to be simple, measurable, attainable, relevant, and limited to a particular time and place. Avoid why questions.

Next, write a prediction that answers your question. This is your hypothesis .

Now that you have a defined population, measure your variable, and obtain data. Don’t forget to write it down in your journal.

Finally, compare your hypothesis with your actual data and write a conclusion about your findings.

These simple and fun steps will help you create great questions that will lead you to find interesting answers and discoveries. But remember, this process not only works for scientific questions but also for daily issues, such as why the car stopped working. You can use it to investigate local environmental problems and provide possible solutions for the benefit of your community and future generations.

You can find more information about this topic in: Ryken, A. E., Otto, P., Pritchard, K., & Owens, K. (2007). Field investigations: Using outdoor environments to foster student learning of scientific processes . Pacific Education Institute. 

Related Content

5 Steps: Experience and Education Needed to Become a Marine Biologist

Find Paid Summer Work in Conservation

How to Get your Dream Job Working in Animal Conservation

Best College for Ecology and Environmental Studies

• Student Resources

Recent Posts

The Wild Rockies and You

From Rainforest to the Rockies: COVID's Turn of Events

How to Get Accepted to the Best Colleges in Ecology and Environmental Studies

Get science-backed answers as you write with Paperpal's Research feature

How to Write a Research Question: Types and Examples 

The first step in any research project is framing the research question. It can be considered the core of any systematic investigation as the research outcomes are tied to asking the right questions. Thus, this primary interrogation point sets the pace for your research as it helps collect relevant and insightful information that ultimately influences your work.   

Typically, the research question guides the stages of inquiry, analysis, and reporting. Depending on the use of quantifiable or quantitative data, research questions are broadly categorized into quantitative or qualitative research questions. Both types of research questions can be used independently or together, considering the overall focus and objectives of your research.  

What is a research question?

A research question is a clear, focused, concise, and arguable question on which your research and writing are centered. 1 It states various aspects of the study, including the population and variables to be studied and the problem the study addresses. These questions also set the boundaries of the study, ensuring cohesion. 

Designing the research question is a dynamic process where the researcher can change or refine the research question as they review related literature and develop a framework for the study. Depending on the scale of your research, the study can include single or multiple research questions. 

A good research question has the following features: 

• It is relevant to the chosen field of study. 
• The question posed is arguable and open for debate, requiring synthesizing and analysis of ideas. 
• It is focused and concisely framed. 
• A feasible solution is possible within the given practical constraint and timeframe. 

A poorly formulated research question poses several risks. 1   

• Researchers can adopt an erroneous design. 
• It can create confusion and hinder the thought process, including developing a clear protocol.  
• It can jeopardize publication efforts.  
• It causes difficulty in determining the relevance of the study findings.  
• It causes difficulty in whether the study fulfils the inclusion criteria for systematic review and meta-analysis. This creates challenges in determining whether additional studies or data collection is needed to answer the question.  
• Readers may fail to understand the objective of the study. This reduces the likelihood of the study being cited by others. 

Now that you know “What is a research question?”, let’s look at the different types of research questions. 

Types of research questions

Depending on the type of research to be done, research questions can be classified broadly into quantitative, qualitative, or mixed-methods studies. Knowing the type of research helps determine the best type of research question that reflects the direction and epistemological underpinnings of your research. 

The structure and wording of quantitative 2 and qualitative research 3 questions differ significantly. The quantitative study looks at causal relationships, whereas the qualitative study aims at exploring a phenomenon. 

• Quantitative research questions:  
• Seeks to investigate social, familial, or educational experiences or processes in a particular context and/or location.  
• Answers ‘how,’ ‘what,’ or ‘why’ questions. 
• Investigates connections, relations, or comparisons between independent and dependent variables. 

Quantitative research questions can be further categorized into descriptive, comparative, and relationship, as explained in the Table below. 

• Qualitative research questions  

Qualitative research questions are adaptable, non-directional, and more flexible. It concerns broad areas of research or more specific areas of study to discover, explain, or explore a phenomenon. These are further classified as follows: 

• Mixed-methods studies  

Mixed-methods studies use both quantitative and qualitative research questions to answer your research question. Mixed methods provide a complete picture than standalone quantitative or qualitative research, as it integrates the benefits of both methods. Mixed methods research is often used in multidisciplinary settings and complex situational or societal research, especially in the behavioral, health, and social science fields. 

What makes a good research question

A good research question should be clear and focused to guide your research. It should synthesize multiple sources to present your unique argument, and should ideally be something that you are interested in. But avoid questions that can be answered in a few factual statements. The following are the main attributes of a good research question. 

• Specific: The research question should not be a fishing expedition performed in the hopes that some new information will be found that will benefit the researcher. The central research question should work with your research problem to keep your work focused. If using multiple questions, they should all tie back to the central aim. 
• Measurable: The research question must be answerable using quantitative and/or qualitative data or from scholarly sources to develop your research question. If such data is impossible to access, it is better to rethink your question. 
• Attainable: Ensure you have enough time and resources to do all research required to answer your question. If it seems you will not be able to gain access to the data you need, consider narrowing down your question to be more specific. 
• You have the expertise 
• You have the equipment and resources 
• Realistic: Developing your research question should be based on initial reading about your topic. It should focus on addressing a problem or gap in the existing knowledge in your field or discipline. 
• Based on some sort of rational physics 
• Can be done in a reasonable time frame 
• Timely: The research question should contribute to an existing and current debate in your field or in society at large. It should produce knowledge that future researchers or practitioners can later build on. 
• Novel 
• Based on current technologies. 
• Important to answer current problems or concerns. 
• Lead to new directions. 
• Important: Your question should have some aspect of originality. Incremental research is as important as exploring disruptive technologies. For example, you can focus on a specific location or explore a new angle. 
• Meaningful whether the answer is “Yes” or “No.” Closed-ended, yes/no questions are too simple to work as good research questions. Such questions do not provide enough scope for robust investigation and discussion. A good research question requires original data, synthesis of multiple sources, and original interpretation and argumentation before providing an answer. 

Steps for developing a good research question

The importance of research questions cannot be understated. When drafting a research question, use the following frameworks to guide the components of your question to ease the process. 4  

• Determine the requirements: Before constructing a good research question, set your research requirements. What is the purpose? Is it descriptive, comparative, or explorative research? Determining the research aim will help you choose the most appropriate topic and word your question appropriately. 
• Select a broad research topic: Identify a broader subject area of interest that requires investigation. Techniques such as brainstorming or concept mapping can help identify relevant connections and themes within a broad research topic. For example, how to learn and help students learn. 
• Perform preliminary investigation: Preliminary research is needed to obtain up-to-date and relevant knowledge on your topic. It also helps identify issues currently being discussed from which information gaps can be identified. 
• Narrow your focus: Narrow the scope and focus of your research to a specific niche. This involves focusing on gaps in existing knowledge or recent literature or extending or complementing the findings of existing literature. Another approach involves constructing strong research questions that challenge your views or knowledge of the area of study (Example: Is learning consistent with the existing learning theory and research). 
• Identify the research problem: Once the research question has been framed, one should evaluate it. This is to realize the importance of the research questions and if there is a need for more revising (Example: How do your beliefs on learning theory and research impact your instructional practices). 

How to write a research question

Those struggling to understand how to write a research question, these simple steps can help you simplify the process of writing a research question. 

Sample Research Questions

The following are some bad and good research question examples 

• Example 1 
• Example 2 

References:  

• Thabane, L., Thomas, T., Ye, C., & Paul, J. (2009). Posing the research question: not so simple.  Canadian Journal of Anesthesia/Journal canadien d’anesthésie ,  56 (1), 71-79. 
• Rutberg, S., & Bouikidis, C. D. (2018). Focusing on the fundamentals: A simplistic differentiation between qualitative and quantitative research.  Nephrology Nursing Journal ,  45 (2), 209-213. 
• Kyngäs, H. (2020). Qualitative research and content analysis.  The application of content analysis in nursing science research , 3-11. 
• Mattick, K., Johnston, J., & de la Croix, A. (2018). How to… write a good research question.  The clinical teacher ,  15 (2), 104-108. 
• Fandino, W. (2019). Formulating a good research question: Pearls and pitfalls.  Indian Journal of Anaesthesia ,  63 (8), 611. 
• Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: a key to evidence-based decisions.  ACP journal club ,  123 (3), A12-A13 

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 21+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Scientific Writing Style Guides Explained
• Ethical Research Practices For Research with Human Subjects
• 8 Most Effective Ways to Increase Motivation for Thesis Writing 
• 6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Transitive and Intransitive Verbs in the World of Research

Language and grammar rules for academic writing, you may also like, academic editing: how to self-edit academic text with..., measuring academic success: definition & strategies for excellence, phd qualifying exam: tips for success , quillbot review: features, pricing, and free alternatives, what is an academic paper types and elements , 9 steps to publish a research paper, what are the different types of research papers, how to make translating academic papers less challenging, 6 tips for post-doc researchers to take their..., presenting research data effectively through tables and figures.

Hypothesis Maker

Ai-powered research hypothesis generator.

• Scientific Research: Generate a hypothesis for your experimental or observational study based on your research question.
• Academic Studies: Formulate a hypothesis for your thesis, dissertation, or academic paper.
• Market Research: Develop a hypothesis for your market research study to understand consumer behavior or market trends.
• Social Science Research: Create a hypothesis for your social science research to explore societal or behavioral patterns.

New & Trending Tools

In-cite ai reference generator, legal text refiner, job search ai assistant.

Research Question Generator

Dublin core, description.

Learning Objects

Output Formats

• Affiliate Program

• UNITED STATES
• 台灣 (TAIWAN)
• TÜRKIYE (TURKEY)
• Academic Editing Services
• - Research Paper
• - Journal Manuscript
• - Dissertation
• - College & University Assignments
• Admissions Editing Services
• - Application Essay
• - Personal Statement
• - Recommendation Letter
• - Cover Letter
• - CV/Resume
• Business Editing Services
• - Business Documents
• - Report & Brochure
• - Website & Blog
• Writer Editing Services
• - Script & Screenplay
• Our Editors
• Client Reviews
• Editing & Proofreading Prices
• Wordvice Points
• Partner Discount
• Plagiarism Checker
• APA Citation Generator
• MLA Citation Generator
• Chicago Citation Generator
• Vancouver Citation Generator
• - APA Style
• - MLA Style
• - Chicago Style
• - Vancouver Style
• Writing & Editing Guide
• Academic Resources
• Admissions Resources

How to Write a Good Research Question (w/ Examples)

What is a Research Question?

A research question is the main question that your study sought or is seeking to answer. A clear research question guides your research paper or thesis and states exactly what you want to find out, giving your work a focus and objective. Learning  how to write a hypothesis or research question is the start to composing any thesis, dissertation, or research paper. It is also one of the most important sections of a research proposal . 

A good research question not only clarifies the writing in your study; it provides your readers with a clear focus and facilitates their understanding of your research topic, as well as outlining your study’s objectives. Before drafting the paper and receiving research paper editing (and usually before performing your study), you should write a concise statement of what this study intends to accomplish or reveal.

Research Question Writing Tips

Listed below are the important characteristics of a good research question:

A good research question should:

• Be clear and provide specific information so readers can easily understand the purpose.
• Be focused in its scope and narrow enough to be addressed in the space allowed by your paper
• Be relevant and concise and express your main ideas in as few words as possible, like a hypothesis.
• Be precise and complex enough that it does not simply answer a closed “yes or no” question, but requires an analysis of arguments and literature prior to its being considered acceptable. 
• Be arguable or testable so that answers to the research question are open to scrutiny and specific questions and counterarguments.

Some of these characteristics might be difficult to understand in the form of a list. Let’s go into more detail about what a research question must do and look at some examples of research questions.

The research question should be specific and focused 

Research questions that are too broad are not suitable to be addressed in a single study. One reason for this can be if there are many factors or variables to consider. In addition, a sample data set that is too large or an experimental timeline that is too long may suggest that the research question is not focused enough.

A specific research question means that the collective data and observations come together to either confirm or deny the chosen hypothesis in a clear manner. If a research question is too vague, then the data might end up creating an alternate research problem or hypothesis that you haven’t addressed in your Introduction section .

The research question should be based on the literature 

An effective research question should be answerable and verifiable based on prior research because an effective scientific study must be placed in the context of a wider academic consensus. This means that conspiracy or fringe theories are not good research paper topics.

Instead, a good research question must extend, examine, and verify the context of your research field. It should fit naturally within the literature and be searchable by other research authors.

References to the literature can be in different citation styles and must be properly formatted according to the guidelines set forth by the publishing journal, university, or academic institution. This includes in-text citations as well as the Reference section . 

The research question should be realistic in time, scope, and budget

There are two main constraints to the research process: timeframe and budget.

A proper research question will include study or experimental procedures that can be executed within a feasible time frame, typically by a graduate doctoral or master’s student or lab technician. Research that requires future technology, expensive resources, or follow-up procedures is problematic.

A researcher’s budget is also a major constraint to performing timely research. Research at many large universities or institutions is publicly funded and is thus accountable to funding restrictions. 

The research question should be in-depth

Research papers, dissertations and theses , and academic journal articles are usually dozens if not hundreds of pages in length.

A good research question or thesis statement must be sufficiently complex to warrant such a length, as it must stand up to the scrutiny of peer review and be reproducible by other scientists and researchers.

Research Question Types

Qualitative and quantitative research are the two major types of research, and it is essential to develop research questions for each type of study. 

Quantitative Research Questions

Quantitative research questions are specific. A typical research question involves the population to be studied, dependent and independent variables, and the research design.

In addition, quantitative research questions connect the research question and the research design. In addition, it is not possible to answer these questions definitively with a “yes” or “no” response. For example, scientific fields such as biology, physics, and chemistry often deal with “states,” in which different quantities, amounts, or velocities drastically alter the relevance of the research.

As a consequence, quantitative research questions do not contain qualitative, categorical, or ordinal qualifiers such as “is,” “are,” “does,” or “does not.”

Categories of quantitative research questions

Qualitative research questions.

In quantitative research, research questions have the potential to relate to broad research areas as well as more specific areas of study. Qualitative research questions are less directional, more flexible, and adaptable compared with their quantitative counterparts. Thus, studies based on these questions tend to focus on “discovering,” “explaining,” “elucidating,” and “exploring.”

Categories of qualitative research questions

Quantitative and qualitative research question examples.

Good and Bad Research Question Examples

Below are some good (and not-so-good) examples of research questions that researchers can use to guide them in crafting their own research questions.

Research Question Example 1

The first research question is too vague in both its independent and dependent variables. There is no specific information on what “exposure” means. Does this refer to comments, likes, engagement, or just how much time is spent on the social media platform?

Second, there is no useful information on what exactly “affected” means. Does the subject’s behavior change in some measurable way? Or does this term refer to another factor such as the user’s emotions?

Research Question Example 2

In this research question, the first example is too simple and not sufficiently complex, making it difficult to assess whether the study answered the question. The author could really only answer this question with a simple “yes” or “no.” Further, the presence of data would not help answer this question more deeply, which is a sure sign of a poorly constructed research topic.

The second research question is specific, complex, and empirically verifiable. One can measure program effectiveness based on metrics such as attendance or grades. Further, “bullying” is made into an empirical, quantitative measurement in the form of recorded disciplinary actions.

Steps for Writing a Research Question

Good research questions are relevant, focused, and meaningful. It can be difficult to come up with a good research question, but there are a few steps you can follow to make it a bit easier.

1. Start with an interesting and relevant topic

Choose a research topic that is interesting but also relevant and aligned with your own country’s culture or your university’s capabilities. Popular academic topics include healthcare and medical-related research. However, if you are attending an engineering school or humanities program, you should obviously choose a research question that pertains to your specific study and major.

Below is an embedded graph of the most popular research fields of study based on publication output according to region. As you can see, healthcare and the basic sciences receive the most funding and earn the highest number of publications. 

2. Do preliminary research  

You can begin doing preliminary research once you have chosen a research topic. Two objectives should be accomplished during this first phase of research. First, you should undertake a preliminary review of related literature to discover issues that scholars and peers are currently discussing. With this method, you show that you are informed about the latest developments in the field.

Secondly, identify knowledge gaps or limitations in your topic by conducting a preliminary literature review . It is possible to later use these gaps to focus your research question after a certain amount of fine-tuning.

3. Narrow your research to determine specific research questions

You can focus on a more specific area of study once you have a good handle on the topic you want to explore. Focusing on recent literature or knowledge gaps is one good option. 

By identifying study limitations in the literature and overlooked areas of study, an author can carve out a good research question. The same is true for choosing research questions that extend or complement existing literature.

4. Evaluate your research question

Make sure you evaluate the research question by asking the following questions:

Is my research question clear?

The resulting data and observations that your study produces should be clear. For quantitative studies, data must be empirical and measurable. For qualitative, the observations should be clearly delineable across categories.

Is my research question focused and specific?

A strong research question should be specific enough that your methodology or testing procedure produces an objective result, not one left to subjective interpretation. Open-ended research questions or those relating to general topics can create ambiguous connections between the results and the aims of the study. 

Is my research question sufficiently complex?

The result of your research should be consequential and substantial (and fall sufficiently within the context of your field) to warrant an academic study. Simply reinforcing or supporting a scientific consensus is superfluous and will likely not be well received by most journal editors.  

Editing Your Research Question

Your research question should be fully formulated well before you begin drafting your research paper. However, you can receive English paper editing and proofreading services at any point in the drafting process. Language editors with expertise in your academic field can assist you with the content and language in your Introduction section or other manuscript sections. And if you need further assistance or information regarding paper compositions, in the meantime, check out our academic resources , which provide dozens of articles and videos on a variety of academic writing and publication topics.

ACRL Framework for Information Literacy Sandbox

A platform and repository for sharing framework materials.

• Contributors

Search form

You are here, research question generator.

Developed for the University of Connecticut's "Research Now!" online curriculum. This worksheet is designed as a tool to narrow a student's topic in order to write a refined research question.

Resource Type(s):

Information literacy frame(s) addressed:.

• Information Creation as Process
• Research as Inquiry
• Searching as Strategic Exploration

Type of Institution:

• Community or Junior College
• Classroom-level

A Systematic Review of Automatic Question Generation for Educational Purposes

• Open access
• Published: 21 November 2019
• Volume 30 , pages 121–204, ( 2020 )

Cite this article

You have full access to this open access article

• Ghader Kurdi   ORCID: orcid.org/0000-0003-1745-5581 1 ,
• Jared Leo 1 ,
• Bijan Parsia 1 ,
• Uli Sattler 1 &
• Salam Al-Emari 2  

45k Accesses

161 Citations

15 Altmetric

Explore all metrics

While exam-style questions are a fundamental educational tool serving a variety of purposes, manual construction of questions is a complex process that requires training, experience, and resources. This, in turn, hinders and slows down the use of educational activities (e.g. providing practice questions) and new advances (e.g. adaptive testing) that require a large pool of questions. To reduce the expenses associated with manual construction of questions and to satisfy the need for a continuous supply of new questions, automatic question generation (AQG) techniques were introduced. This review extends a previous review on AQG literature that has been published up to late 2014. It includes 93 papers that were between 2015 and early 2019 and tackle the automatic generation of questions for educational purposes. The aims of this review are to: provide an overview of the AQG community and its activities, summarise the current trends and advances in AQG, highlight the changes that the area has undergone in the recent years, and suggest areas for improvement and future opportunities for AQG. Similar to what was found previously, there is little focus in the current literature on generating questions of controlled difficulty, enriching question forms and structures, automating template construction, improving presentation, and generating feedback. Our findings also suggest the need to further improve experimental reporting, harmonise evaluation metrics, and investigate other evaluation methods that are more feasible.

Similar content being viewed by others

The Promises and Challenges of Artificial Intelligence for Teachers: a Systematic Review of Research

Artificial intelligence in higher education: the state of the field

An automated essay scoring systems: a systematic literature review

Avoid common mistakes on your manuscript.

Introduction

Exam-style questions are a fundamental educational tool serving a variety of purposes. In addition to their role as an assessment instrument, questions have the potential to influence student learning. According to Thalheimer ( 2003 ), some of the benefits of using questions are: 1) offering the opportunity to practice retrieving information from memory; 2) providing learners with feedback about their misconceptions; 3) focusing learners’ attention on the important learning material; 4) reinforcing learning by repeating core concepts; and 5) motivating learners to engage in learning activities (e.g. reading and discussing). Despite these benefits, manual question construction is a challenging task that requires training, experience, and resources. Several published analyses of real exam questions (mostly multiple choice questions (MCQs)) (Hansen and Dexter 1997 ; Tarrant et al. 2006 ; Hingorjo and Jaleel 2012 ; Rush et al. 2016 ) demonstrate their poor quality, which Tarrant et al. ( 2006 ) attributed to a lack of training in assessment development. This challenge is augmented further by the need to replace assessment questions consistently to ensure their validity, since their value will decrease or be lost after a few rounds of usage (due to being shared between test takers), as well as the rise of e-learning technologies, such as massive open online courses (MOOCs) and adaptive learning, which require a larger pool of questions.

Automatic question generation (AQG) techniques emerged as a solution to the challenges facing test developers in constructing a large number of good quality questions. AQG is concerned with the construction of algorithms for producing questions from knowledge sources, which can be either structured (e.g. knowledge bases (KBs) or unstructured (e.g. text)). As Alsubait ( 2015 ) discussed, research on AQG goes back to the 70’s. Nowadays, AQG is gaining further importance with the rise of MOOCs and other e-learning technologies (Qayyum and Zawacki-Richter 2018 ; Gaebel et al. 2014 ; Goldbach and Hamza-Lup 2017 ).

In what follows, we outline some potential benefits that one might expect from successful automatic generation of questions. AQG can reduce the cost (in terms of both money and effort) of question construction which, in turn, enables educators to spend more time on other important instructional activities. In addition to resource saving, having a large number of good-quality questions enables the enrichment of the teaching process with additional activities such as adaptive testing (Vie et al. 2017 ), which aims to adapt learning to student knowledge and needs, as well as drill and practice exercises (Lim et al. 2012 ). Finally, being able to automatically control question characteristics, such as question difficulty and cognitive level, can inform the construction of good quality tests with particular requirements.

Although the focus of this review is education, the applications of question generation (QG) are not limited to education and assessment. Questions are also generated for other purposes, such as validation of knowledge bases, development of conversational agents, and development of question answering or machine reading comprehension systems, where questions are used for training and testing.

This review extends a previous systematic review on AQG (Alsubait 2015 ), which covers the literature up to the end of 2014. Given the large amount of research that has been published since Alsubait’s review was conducted (93 papers over a four year period compared to 81 papers over the preceding 45-year period), an extension of Alsubait’s review is reasonable at this stage. To capture the recent developments in the field, we review the literature on AQG from 2015 to early 2019. We take Alsubait’s review as a starting point and extend the methodology in a number of ways (e.g. additional review questions and exclusion criteria), as will be described in the sections titled “ Review Objective ” and “ Review Method ”. The contribution of this review is in providing researchers interested in the field with the following:

a comprehensive summary of the recent AQG approaches;

an analysis of the state of the field focusing on differences between the pre- and post-2014 periods;

a summary of challenges and future directions; and

an extensive reference to the relevant literature.

Summary of Previous Reviews

There have been six published reviews on the AQG literature. The reviews reported by Le et al. 2014 , Kaur and Bathla 2015 , Alsubait 2015 and Rakangor and Ghodasara ( 2015 ) cover the literature that has been published up to late 2014 while those reported by Ch and Saha ( 2018 ) and Papasalouros and Chatzigiannakou ( 2018 ) cover the literature that has been published up to late 2018. Out of these, the most comprehensive review is Alsubait’s, which includes 81 papers (65 distinct studies) that were identified using a systematic procedure. The other reviews were selective and only cover a small subset of the AQG literature. Of interest, due to it being a systematic review and due to the overlap in timing with our review, is the review developed by Ch and Saha ( 2018 ). However, their review is not as rigorous as ours, as theirs only focuses on automatic generation of MCQs using text as input. In addition, essential details about the review procedure, such as the search queries used for each electronic database and the resultant number of papers, are not reported. In addition, several related studies found in other reviews on AQG are not included.

Findings of Alsubait’s Review

In this section, we concentrate on summarising the main results of Alsubait’s systematic review, due to its being the only comprehensive review. We do so by elaborating on interesting trends and speculating about the reasons for those trends, as well as highlighting limitations observed in the AQG literature.

Alsubait characterised AQG studies along the following dimensions: 1) purpose of generating questions, 2) domain, 3) knowledge sources, 4) generation method, 5) question type, 6) response format, and 7) evaluation.

The results of the review and the most prevalent categories within each dimension are summarised in Table  1 . As can be seen in Table  1 , generating questions for a specific domain is more prevalent than generating domain-unspecific questions. The most investigated domain is language learning (20 studies), followed by mathematics and medicine (four studies each). Note that, for these three domains, there are large standardised tests developed by professional organisations (e.g. Test of English as a Foreign Language (TOEFL), International English Language Testing System (IELTS) and Test of English for International Communication (TOEIC) for language, Scholastic Aptitude Test (SAT) for mathematics and board examinations for medicine). These tests require a continuous supply of new questions. We believe that this is one reason for the interest in generating questions for these domains. We also attribute the interest in the language learning domain to the ease of generating language questions, relative to questions belonging to other domains. Generating language questions is easier than generating other types of questions for two reasons: 1) the ease of adopting text from a variety of publicly available resources (e.g. a large number of general or specialised textual resources can be used for reading comprehension (RC)) and 2) the availability of natural language processing (NLP) tools for shallow understanding of text (e.g. part of speech (POS) tagging) with an acceptable performance, which is often sufficient for generating language questions. To illustrate, in Chen et al. ( 2006 ), the distractors accompanying grammar questions are generated by changing the verb form of the key (e.g. “write”, “written”, and “wrote” are distractors while “writing” is the key). Another plausible reason for interest in questions on medicine is the availability of NLP tools (e.g. named entity recognisers and co-reference resolvers) for processing medical text. There are also publicly available knowledge bases, such as UMLS (Bodenreider 2004 ) and SNOMED-CT (Donnelly 2006 ), that are utilised in different tasks such as text annotation and distractor generation. The other investigated domains are analytical reasoning, geometry, history, logic, programming, relational databases, and science (one study each).

With regard to knowledge sources, the most commonly used source for question generation is text (Table  1 ). A similar trend was also found by Rakangor and Ghodasara ( 2015 ). Note that 19 text-based approaches, out of the 38 text-based approaches identified by Alsubait ( 2015 ), tackle the generation of questions for the language learning domain, both free response (FR) and multiple choice (MC). Out of the remaining 19 studies, only five focus on generating MCQs. To do so, they incorporate additional inputs such as WordNet (Miller et al. 1990 ), thesaurus, or textual corpora. By and large, the challenge in the case of MCQs is distractor generation. Despite using text for generating language questions, where distractors can be generated using simple strategies such as selecting words having a particular POS or other syntactic properties, text often does not incorporate distractors, so external, structured knowledge sources are needed to find what is true and what is similar. On the other hand, eight ontology-based approaches are centred on generating MCQs and only three focus on FR questions.

Simple factual wh-questions (i.e. where the answers are short facts that are explicitly mentioned in the input) and gap-fill questions (also known as fill-in-the-blank or cloze questions) are the most generated types of questions with the majority of them, 17 and 15 respectively, being generated from text. The prevalence of these questions is expected because they are common in language learning assessment. In addition, these two types require relatively little effort to construct, especially when they are not accompanied by distractors. In gap-fill questions, there are no concerns about the linguistic aspects (e.g. grammaticality) because the stem is constructed by only removing a word or a phrase from a segment of text. The stem of a wh-question is constructed by removing the answer from the sentence, selecting an appropriate wh-word, and rearranging words to form a question. Other types of questions such as mathematical word problems, Jeopardy-style questions, Footnote 1 and medical case-based questions (CBQs) require more effort in choosing the stem content and verbalisation. Another related observation we made is that the types of questions generated from ontologies are more varied than the types of questions generated from text.

Limitations observed by Alsubait ( 2015 ) include the limited research on controlling the difficulty of generated questions and on generating informative feedback. Existing difficulty models are either not validated or only applicable to a specific type of question (Alsubait 2015 ). Regarding feedback (i.e. an explanation for the correctness/incorrectness of the answer), only three studies generate feedback along with the questions. Even then, the feedback is used to motivate students to try again or to provide extra reading material without explaining why the selected answer is correct/incorrect. Ungrammaticality is another notable problem with auto-generated questions, especially in approaches that apply syntactic transformations of sentences (Alsubait 2015 ). For example, 36.7% and 39.5% of questions generated in the work of Heilman and Smith ( 2009 ) were rated by reviewers as ungrammatical and nonsensical, respectively. Another limitation related to approaches to generating questions from ontologies is the use of experimental ontologies for evaluation, neglecting the value of using existing, probably large, ontologies. Various issues can arise if existing ontologies are used, which in turn provide further opportunities to enhance the quality of generated questions and the ontologies used for generation.

Review Objective

The goal of this review is to provide a comprehensive view of the AQG field since 2015. Following and extending the schema presented by Alsubait ( 2015 ) (Table  1 ), we have structured our review around the following four objectives and their related questions. Questions marked with an asterisk “*” are those proposed by Alsubait ( 2015 ). Questions under the first three objectives (except question 5 under OBJ3) are used to guide data extraction. The others are analytical questions to be answered based on extracted results.

Providing an overview of the AQG community and its activities

What is the rate of publication?*

What types of papers are published in the area?

Where is research published?

Who are the active research groups in the field?*

Summarising current QG approaches

What is the purpose of QG?*

What method is applied?*

What tasks related to question generation are considered?

What type of input is used?*

Is it designed for a specific domain? For which domain?*

What type of questions are generated?* (i.e., question format and answer format)

What is the language of the questions?

Does it generate feedback?*

Is difficulty of questions controlled?*

Does it consider verbalisation (i.e. presentation improvements)?

Identifying the gold-standard performance in AQG

Are there any available sources or standard datasets for performance comparison?

What types of evaluation are applied to QG approaches?*

What properties of questions are evaluated? Footnote 2 and What metrics are used for their measurement?

How does the generation approach perform?

What is the gold-standard performance?

Tracking the evolution of AQG since Alsubait’s review

Has there been any progress on feedback generation?

Has there been progress on generating questions with controlled difficulty?

Has there been progress on enhancing the naturalness of questions (i.e. verbalisation)?

One of our motivations for pursuing these objectives is to provide members of the AQG community with a reference to facilitate decisions such as what resources to use, whom to compare to, and where to publish. As we mentioned in the  Summary of Previous Reviews , Alsubait ( 2015 ) highlighted a number of concerns related to the quality of generated questions, difficulty models, and the evaluation of questions. We were motivated to know whether these concerns have been addressed. Furthermore, while reviewing some of the AQG literature, we made some observations about the simplicity of generated questions and about the reporting being insufficient and heterogeneous. We want to know whether these issues are universal across the AQG literature.

Review Method

We followed the systematic review procedure explained in (Kitchenham and Charters 2007 ; Boland et al. 2013 ).

Inclusion and Exclusion Criteria

We included studies that tackle the generation of questions for educational purposes (e.g. tutoring systems, assessment, and self-assessment) without any restriction on domains or question types. We adopted the exclusion criteria used in Alsubait ( 2015 ) (1 to 5) and added additional exclusion criteria (6 to 13). A paper is excluded if:

it is not in English

it presents work in progress only and does not provide a sufficient description of how the questions are generated

it presents a QG approach that is based mainly on a template and questions are generated by substituting template slots with numerals or with a set of randomly predefined values

it focuses on question answering rather than question generation

it presents an automatic mechanism to deliver assessments, rather than generating assessment questions

it presents an automatic mechanism to assemble exams or to adaptively select questions from a question bank

it presents an approach for predicting the difficulty of human-authored questions

it presents a QG approach for purposes other than those related to education (e.g. training of question answering systems, dialogue systems)

it does not include an evaluation of the generated questions

it is an extension of a paper published before 2015 and no changes were made to the question generation approach

it is a secondary study (i.e. literature review)

it is not peer-reviewed (e.g. theses, presentations and technical reports)

its full text is not available (through the University of Manchester Library website, Google or Google scholar).

Search Strategy

Data sources.

Six data sources were used, five of which were electronic databases (ERIC, ACM, IEEE, INSPEC and Science Direct), which were determined by Alsubait ( 2015 ) to have good coverage of the AQG literature. We also searched the International Journal of Artificial Intelligence in Education (AIED) and the proceedings of the International Conference on Artificial Intelligence in Education for 2015, 2017, and 2018 due to their AQG publication record.

We obtained additional papers by examining the reference lists of, and the citations to, AQG papers we reviewed (known as “snowballing”). The citations to a paper were identified by searching for the paper using Google Scholar, then clicking on the “cited by” option that appears under the name of the paper. We performed this for every paper on AQG, regardless of whether we had decided to include it, to ensure that we captured all the relevant papers. That is to say, even if a paper was excluded because it met some of the exclusion criteria (1-3 and 8-13), it is still possible that it refers to, or is referred to by, relevant papers.

We used the reviews reported by Ch and Saha ( 2018 ) and Papasalouros and Chatzigiannakou ( 2018 ) as a “sanity check” to evaluate the comprehensiveness of our search strategy. We exported all the literature published between 2015 and 2018 included in the work of Ch and Saha ( 2018 ) and Papasalouros and Chatzigiannakou ( 2018 ) and checked whether they were included in our results (both search results and snowballing results).

Search Queries

We used the keywords “question” and “generation” to search for relevant papers. Actual search queries used for each of the databases are provided in the Appendix under “ Search Queries ”. We decided on these queries after experimenting with different combinations of keywords and operators provided by each database and looking at the ratio between relevant and irrelevant results in the first few pages (sorted by relevance). To ensure that recall was not compromised, we checked whether relevant results returned using different versions of each search query were still captured by the selected version.

The search results were exported to comma-separated values (CSV) files. Two reviewers then looked independently at the titles and abstracts to decide on inclusion or exclusion. The reviewers skimmed the paper if they were not able to make a decision based on the title and abstract. Note that, at this phase, it was not possible to assess whether all papers had satisfied the exclusion criteria 2, 3, 8, 9, and 10. Because of this, the final decision was made after reading the full text as described next.

To judge whether a paper’s purpose was related to education, we considered the title, abstract, introduction, and conclusion sections. Papers that mentioned many potential purposes for generating questions, but did not state which one was the focus, were excluded. If the paper mentioned only educational applications of QG, we assumed that its purpose was related to education, even without a clear purpose statement. Similarly, if the paper mentioned only one application, we assumed that was its focus.

Concerning evaluation, papers that evaluated the usability of a system that had a QG functionality, without evaluating the quality of generated questions, were excluded. In addition, in cases where we found multiple papers by the same author(s) reporting the same generation approach, even if some did not cover evaluation, all of the papers were included but counted as one study in our analyses.

Lastly, because the final decision on inclusion/exclusion sometimes changed after reading the full paper, agreement between the two reviewers was checked after the full paper had been read and the final decision had been made. However, a check was also made to ensure that the inclusion/exclusion criteria were interpreted in the same way. Cases of disagreement were resolved through discussion.

Data Extraction

Guided by the questions presented in the “ Review Objective ” section, we designed a specific data extraction form. Two reviewers independently extracted data related to the included studies. As mentioned above, different papers that related to the same study were represented as one entry. Agreement for data extraction was checked and cases of disagreement were discussed to reach a consensus.

Papers that had at least one shared author were grouped together if one of the following criteria were met:

they reported on different evaluations of the same generation approach;

they reported on applying the same generation approach to different sources or domains;

one of the papers introduced an additional feature of the generation approach such as difficulty prediction or generating distractors without changing the initial generation procedure.

The extracted data were analysed using a code written in R markdown. Footnote 3

Quality Assessment

Since one of the main objectives of this review is to identify the gold standard performance, we were interested in the quality of the evaluation approaches. To assess this, we used the criteria presented in Table  2 which were selected from existing checklists (Downs and Black 1998 ; Reisch et al. 1989 ; Critical Appraisal Skills Programme 2018 ), with some criteria being adapted to fit specific aspects of research on AQG. The quality assessment was conducted after reading a paper and filling in the data extraction form.

In what follows, we describe the individual criteria (Q1-Q9 presented in Table  2 ) that we considered when deciding if a study satisfied said criteria. Three responses are used when scoring the criteria: “yes”, “no” and “not specified”. The “not specified” response is used when either there is no information present to support the criteria, or when there is not enough information present to distinguish between a “yes” or “no” response.

Q1-Q4 are concerned with the quality of reporting on participant information, Q5-Q7 are concerned with the quality of reporting on the question samples, and Q8 and Q9 describe the evaluative measures used to assess the outcomes of the studies.

When a study reports the exact number of participants (e.g. experts, students, employees, etc.) used in the study, Q1 scores a “yes”. Otherwise, it scores a “no”. For example, the passage “20 students were recruited to participate in an exam …” would result in a “yes”, whereas “a group of students were recruited to participate in an exam …” would result in a “no”.

Q2 requires the reporting of demographic characteristics supporting the suitability of the participants for the task. Depending on the category of participant, relevant demographic information is required to score a “yes”. Studies that do not specify relevant information score a “no”. By means of examples, in studies relying on expert reviews, those that include information on teaching experience or the proficiency level of reviewers would receive a “yes”, while in studies relying on mock exams, those that include information about grade level or proficiency level of test takers would also receive a “yes”. Studies reporting that the evaluation was conducted by reviewers, instructors, students, or co-workers without providing any additional information about the suitability of the participants for the task would be considered neglectful of Q2 and score a “no”.

For a study to score “yes” for Q3, it must provide specific information on how participants were selected/recruited, otherwise it receives a score of “no”. This includes information on whether the participants were paid for their work or were volunteers. For example, the passage “7th grade biology students were recruited from a local school.” would receive a score of “no” because it is not clear whether or not they were paid for their work. However, a study that reports “Student volunteers were recruited from a local school …” or “Employees from company X were employed for n hours to take part in our study… they were rewarded for their services with Amazon vouchers worth $n” would receive a “yes”.

To score “yes” for Q4, two conditions must be met: the study must 1) score “yes” for both Q2 and Q3 and 2) only use participants that are suitable for the task at hand. Studies that fail to meet the first condition score “not specified” while those that fail to meet the second condition score “no”. Regarding the suitability of participants, we consider, as an example, native Chinese speakers suitable for evaluating the correctness and plausibility of options generated for Chinese gap-fill questions. As another example, we consider Amazon Mechanical Turk (AMT) co-workers unsuitable for evaluating the difficulty of domain-specific questions (e.g. mathematical questions).

When a study reports the exact number of questions used in the experimentation or evaluation stage, Q5 receives a score of “yes”, otherwise it receives a score of “no”. To demonstrate, consider the following examples. A study reporting “25 of the 100 generated questions were used in our evaluation …” would receive a score of “yes”. However, if a study made a claim such as “Around half of the generated questions were used …”, it would receive a score of “no”.

Q6a requires that the sampling strategy be not only reported (e.g. random, proportionate stratification, disproportionate stratification, etc.) but also justified to receive a “yes”, otherwise, it receives a score of “no”. To demonstrate, if a study only reports that “We sampled 20 questions from each template …” would receive a score of “no” since no justification as to why the stratified sampling procedure was used is provided. However, if it was to also add “We sampled 20 questions from each template to ensure template balance in discussions about the quality of generated questions …” then this would be considered as a suitable justification and would warrant a score of “yes”. Similarly, Q6b requires that the sample size be both reported and justified.

Our decision regarding Q7 takes into account the following: 1) responses to Q6a (i.e. a study can only score “yes” if the score to Q6a is “yes”, otherwise, the score would be “not specified”) and 2) representativeness of the population. Using random sampling is, in most cases, sufficient to score “yes” for Q7. However, if multiple types of questions are generated (e.g. different templates or different difficulty levels), stratified sampling is more appropriate in cases in which the distribution of questions is skewed.

Q8 considers whether the authors provide a description, a definition, or a mathematical formula for the evaluation measures they used as well as a description of the coding system (if applicable). If so, then the study receives a score of “yes” for Q8, otherwise it receives a score of “no”.

Q9 is concerned with whether questions were evaluated by multiple reviewers and whether measures of the agreement (e.g., Cohen’s kappa or percentage of agreement) were reported. For example, studies reporting information similar to “all questions were double-rated and inter-rater agreement was computed…” receive a score of “yes”, whereas studies reporting information similar to “Each question was rated by one reviewer…” receive a score of “no” .

To assess inter-rater reliability, this activity was performed by two reviewers (the first and second authors), who are proficient in the field of AQG, independently on an exploratory random sample of 27 studies. Footnote 4 The percentage of agreement and Cohen’s kappa were used to measure inter-rater reliability for Q1-Q9. The percentage of agreement ranged from 73% to 100%, while Cohen’s kappa was above .72 for Q1-Q5, demonstrating “substantial to almost perfect agreement”, and equal to 0.42 for Q9, Footnote 5

Results and Discussion

Search and screening results.

Searching the databases and AIED resulted in 2,012 papers and we checked 974. Footnote 7 The difference is due to ACM which provided 1,265 results and we only checked the first 200 results (sorted by relevance) because we found that subsequent results became irrelevant. Out of the search results, 122 papers were considered relevant after looking at their titles and abstracts. After removing duplicates, 89 papers remained. This set was further reduced to 36 papers after reading the full text of the papers. Checking related work sections and the reference lists identified 169 further papers (after removing duplicates). After we read their full texts, we found 46 to satisfy our inclusion criteria. Among those 46, 15 were captured by the initial search. Tracking citations using Google Scholar provided 204 papers (after removing duplicates). After reading their full text, 49 were found to satisfy our inclusion criteria. Among those 49, 14 were captured by the initial search. The search results are outlined in Table  3 . The final number of included papers was 93 (72 studies after grouping papers as described before). In total, the database search identified 36 papers while the other sources identified 57. Although the number of papers identified through other sources was large, many of them were variants of papers already included in the review.

The most common reasons for excluding papers on AQG were that the purpose of the generation was not related to education or there was no evaluation. Details of papers that were excluded after reading their full text are in the Appendix under “ Excluded Studies ”.

Data Extraction Results

In this section, we provide our results and outline commonalities and differences with Alsubait’s results (highlighted in the “ Findings of Alsubait’s Review ” section). The results are presented in the same order as our research questions. The main characteristics of the reviewed literature can be found in the Appendix under “ Summary of Included Studies ”.

Rate of Publication

The distribution of publications by year is presented in Fig.  1 . Putting this together with the results reported by Alsubait ( 2015 ), we notice a strong increase in publication starting from 2011. We also note that there were three workshops on QG Footnote 8 in 2008, 2009, and 2010, respectively, with one being accompanied by a shared task (Rus et al. 2012 ). We speculate that the increase starting from 2011 is because workshops on QG have drawn researchers’ attention to the field, although the participation rate in the shared task was low (only five groups participated). The increase also coincides with the rise of MOOCs and the launch of major MOOC providers (Udacity, Udemy, Coursera and edX, which all started up in 2012 (Baturay 2015 )) which provides another reason for the increasing interest in AQG. This interest was further boosted from 2015. In addition to the above speculations, it is important to mention that QG is closely related to other areas such as NLP and the Semantic Web. Being more mature and providing methods and tools that perform well have had an effect on the quantity and quality of research in QG. Note that these results are only related to question generation studies that focus on educational purposes and that there is a large volume of studies investigating question generation for other applications as mentioned in the “ Search and Screening Results ” section.

Publications per year

Types of Papers and Publication Venues

Of the papers published in the period covered by this review, conference papers constitute the majority (44 papers), followed by journal articles (32 papers) and workshop papers (17 papers). This is similar to the results of Alsubait ( 2015 ) with 34 conference papers, 22 journal papers, 13 workshop papers, and 12 other types of papers, including books or book chapters as well as technical reports and theses. In the Appendix, under “ Publication Venues ”, we list journals, conferences, and workshops that published at least two of the papers included in either of the reviews.

Research Groups

Overall, 358 researchers are working in the area (168 identified in Alsubait’s review and 205 identified in this review with 15 researchers in common). The majority of researchers have only one publication. In Appendix “ Active Research Groups ”, we present the 13 active groups defined as having more than two publications in the period of both reviews. Of the 174 papers identified in both reviews, 64 were published by these groups. This shows that, besides the increased activities in the study of AQG, the community is also growing.

Purpose of Question Generation

Similar to the results of Alsubait’s review (Table  1 ), the main purpose of generating questions is to use them as assessment instruments (Table  4 ). Questions are also generated for other purposes, such as to be employed in tutoring or self-assisted learning systems. Generated questions are still used in experimental settings and only Zavala and Mendoza ( 2018 ) have reported their use in a class setting, in which the generator is used to generate quizzes for several courses and to generate assignments for students.

Generation Methods

Methods of generating questions have been classified in the literature (Yao et al. 2012 ) as follows: 1) syntax-based, 2) semantic-based, and 3) template-based. Syntax-based approaches operate on the syntax of the input (e.g. syntactic tree of text) to generate questions. Semantic-based approaches operate on a deeper level (e.g. is-a or other semantic relations). Template-based approaches use templates consisting of fixed text and some placeholders that are populated from the input. Alsubait ( 2015 ) extended this classification to include two more categories: 4) rule-based and 5) schema-based. The main characteristic of rule-based approaches, as defined by Alsubait ( 2015 ), is the use of rule-based knowledge sources to generate questions that assess understanding of the important rules of the domain. As this definition implies that these methods require a deep understanding (beyond syntactic understanding), we believe that this category falls under the semantic-based category. However, we define the rule-based approach differently, as will be seen below. Regarding the fifth category, according to Alsubait ( 2015 ), schemas are similar to templates but are more abstract. They provide a grouping of templates that represent variants of the same problem. We regard this distinction between template and schema as unclear. Therefore, we restrict our classification to the template-based category regardless of how abstract the templates are.

In what follows, we extend and re-organise the classification proposed by Yao et al. ( 2012 ) and extended by Alsubait ( 2015 ). This is due to our belief that there are two relevant dimensions that are not captured by the existing classification of different generation approaches: 1) the level of understanding of the input required by the generation approach and 2) the procedure for transforming the input into questions. We describe our new classification, characterise each category and give examples of features that we have used to place a method within these categories. Note that these categories are not mutually exclusive.

Level of understanding

Syntactic: Syntax-based approaches leverage syntactic features of the input, such as POS or parse-tree dependency relations, to guide question generation. These approaches do not require understanding of the semantics of the input in use (i.e. entities and their meaning). For example, approaches that select distractors based on their POS are classified as syntax-based.

Semantic: Semantic-based approaches require a deeper understanding of the input, beyond lexical and syntactic understanding. The information that these approaches use are not necessarily explicit in the input (i.e. they may require reasoning to be extracted). In most cases, this requires the use of additional knowledge sources (e.g., taxonomies, ontologies, or other such sources). As an example, approaches that use either contextual similarity or feature-based similarity to select distractors are classified as being semantic-based.

Procedure of transformation

Template: Questions are generated with the use of templates. Templates define the surface structure of the questions using fixed text and placeholders that are substituted with values to generate questions. Templates also specify the features of the entities (either syntactic, semantic, or both), that can replace the placeholders.

Rule: Questions are generated with the use of rules. Rules often accompany approaches using text as input. Typically, approaches utilising rules annotate sentences with syntactic and/or semantic information. They then use these annotations to match the input to a pattern specified in the rules. These rules specify how to select a suitable question type (e.g. selecting suitable wh-words) and how to manipulate the input to construct questions (e.g. converting sentences into questions).

Statistical methods: This is where question transformation is learned from training data. For example, in Gao et al. ( 2018 ), question generation has been dealt with as a sequence-to-sequence prediction problem in which, given a segment of text (usually a sentence), the question generator forms a sequence of text representing a question (using the probabilities of co-occurrence that are learned from the training data). Training data has also been used in Kumar et al. ( 2015b ) for predicting which word(s) in the input sentence is/are to be replaced by a gap (in gap-fill questions).

Regarding the level of understanding, 60 papers rely on semantic information and only ten approaches rely only on syntactic information. All except three of the ten syntactic approaches (Das and Majumder 2017 ; Kaur and Singh 2017 ; Kusuma and Alhamri 2018 ) tackle the generation of language questions. In addition, templates are more popular than rules and statistical methods, with 27 papers reporting the use of templates, compared to 16 and nine for rules and statistical methods, respectively. Each of these three approaches has its advantages and disadvantages. In terms of cost, all three approaches are considered expensive. Templates and rules require manual construction, while learning from data often requires a large amount of annotated data which is unavailable in many specific domains. Additionally, questions generated by rules and statistical methods are very similar to the input (e.g. sentences used for generation), while templates allow the generating of questions that differ from the surface structure of the input, in the use of words for example. However, questions generated from templates are limited in terms of their linguistic diversity. Note that some of the papers were classified as not having a method of transforming the input into questions because they only focused on distractor generation or gap-fill questions for which the stem is the same input statement with a word or a phrase being removed. Readers interested in studies that belong to a specific approach are referred to the “ Summary of Included Studies ” in the Appendix.

Generation Tasks

Tasks involved in question generation are explained below. We grouped the tasks into the stages of preprocessing, question construction, and post-processing. For each task, we provide a brief description, mention its role in the generation process, and summarise different approaches that have been applied in the literature. The “ Summary of Included Studies ” in the Appendix shows which tasks have been tackled in each study.

Preprocessing

Two types of preprocessing are involved: 1) standard preprocessing and 2) QG-specific preprocessing. Standard preprocessing is common to various NLP tasks and is used to prepare the input for upcoming tasks; it involves segmentation, sentence splitting, tokenisation, POS tagging, and coreference resolution. In some cases, it also involves named entity recognition (NER) and relation extraction (RE). The aim of QG-specific preprocessing is to make or select inputs that are more suitable for generating questions. In the reviewed literature, three types of QG-specific preprocessing are employed:

Sentence simplification: This is employed in some text-based approaches (Liu et al. 2017 ; Majumder and Saha 2015 ; Patra and Saha 2018b ). Complex sentences, usually sentences with appositions or sentences joined with conjunctions, are converted into simple sentences to ease upcoming tasks. For example, Patra and Saha ( 2018b ) reported that Wikipedia sentences are long and contain multiple objects; simplifying these sentences facilitates triplet extraction (where triples are used later for generating questions). This task was carried out by using sentence simplification rules (Liu et al. 2017 ) and relying on parse-tree dependencies (Majumder and Saha 2015 ; Patra and Saha 2018b ).

Sentence classification: In this task, sentences are classified into categories, which is, according to Mazidi and Tarau ( 2016a ) and Mazidi and Tarau ( 2016b ), a key to determining the type of question to be asked about the sentence. This classification was carried out by analysing POS and dependency labels, as in Mazidi and Tarau ( 2016a ) and Mazidi and Tarau ( 2016b ) or by using a machine learning (ML) model and a set of rules, as in Basuki and Kusuma ( 2018 ). For example, in Mazidi and Tarau( 2016a , 2016b ), the pattern “S-V-acomp” is an adjectival complement that describes the subject and is therefore matched to the question template “Indicate properties or characteristics of S?”

Content selection: As the number of questions in examinations is limited, the goal of this task is to determine important content, such as sentences, parts of sentences, or concepts, about which to generate questions. In the reviewed literature, the majority approach is to generate all possible questions and leave the task of selecting important questions to exam designers. However, in some settings such as self-assessment and self-learning environments, in which questions are generated “on the fly”, leaving the selection to exam designers is not feasible.

Content selection was of interest for those approaches that utilise text more than for those that utilise structured knowledge sources. Several characterisations of important sentences and approaches for their selection have been proposed in the reviewed literature which we summarise in the following paragraphs.

Huang and He ( 2016 ) defined three characteristics for selecting sentences that are important for reading assessment and propose metrics for their measurement: keyness (containing the key meaning of the text), completeness (spreading over different paragraphs to ensure that test-takers grasp the text fully), and independence (covering different aspects of text content). Olney et al. ( 2017 ) selected sentences that: 1) are well connected to the discourse (same as completeness) and 2) contain specific discourse relations. Other researchers have focused on selecting topically important sentences. To that end, Kumar et al. ( 2015b ) selected sentences that contain concepts and topics from an educational textbook, while Kumar et al. ( 2015a ) and Majumder and Saha ( 2015 ) used topic modelling to identify topics and then rank sentences based on topic distribution. Park et al. ( 2018 ) took another approach by projecting the input document and sentences within it into the same n-dimensional vector space and then selecting sentences that are similar to the document, assuming that such sentences best express the topic or the essence of the document. Other approaches selected sentences by checking the occurrence of, or measuring the similarity to, a reference set of patterns under the assumption that these sentences convey similar information to sentences used to extract patterns (Majumder and Saha 2015 ; Das and Majumder 2017 ). Others (Shah et al. 2017 ; Zhang and Takuma 2015 ) filtered sentences that are insufficient on their own to make valid questions, such as sentences starting with discourse connectives (e.g. thus, also, so, etc.) as in Majumder and Saha ( 2015 ).

Still other approaches to content selection are more specific and are informed by the type of question to be generated. For example, the purpose of the study reported in Susanti et al. ( 2015 ) is to generate “closest-in-meaning vocabulary questions” Footnote 9 which involve selecting a text snippet from the Internet that contains the target word, while making sure that the word has the same sense in both the input and retrieved sentences. To this end, the retrieved text was scored on the basis of metrics such as the number of query words that appear in the text.

With regard to content selection from structured knowledge bases, only one study focuses on this task. Rocha and Zucker ( 2018 ) used DBpedia to generate questions along with external ontologies; the ontologies describe educational standards according to which DBpedia content was selected for use in question generation.

Question Construction

This is the main task and involves different processes based on the type of questions to be generated and their response format. Note that some studies only focus on generating partial questions (only stem or distractors). The processes involved in question construction are as follows:

Stem and correct answer generation: These two processes are often carried out together, using templates, rules, or statistical methods, as mentioned in the “ Generation Methods ” Section. Subprocesses involved are:

transforming assertive sentences into interrogative ones (when the input is text);

determination of question type (i.e. selecting suitable wh-word or template); and

selection of gap position (relevant to gap-fill questions).

Incorrect options (i.e. distractor) generation: Distractor generation is a very important task in MCQ generation since distractors influence question quality. Several strategies have been used to generate distractors. Among these are selection of distractors based on word frequency (i.e. the number of times distractors appear in a corpus is similar to the key) (Jiang and Lee 2017 ), POS (Soonklang and Muangon 2017 ; Susanti et al. 2015 ; Satria and Tokunaga 2017a , 2017b ; Jiang and Lee 2017 ), or co-occurrence with the key (Jiang and Lee 2017 ). A dominant approach is the selection of distractors based on their similarity to the key, using different notions of similarity, such as syntax-based similarity (i.e. similar POS, similar letters) (Kumar et al. 2015b ; Satria and Tokunaga 2017a , 2017b ; Jiang and Lee 2017 ), feature-based similarity (Wita et al. 2018 ; Majumder and Saha 2015 ; Patra and Saha 2018a , 2018b ; Alsubait et al. 2016 ; Leo et al. 2019 ), or contextual similarity (Afzal 2015 ; Kumar et al. 2015a , 2015b ; Yaneva et al. 2018 ; Shah et al. 2017 ; Jiang and Lee 2017 ). Some studies (Lopetegui et al. 2015 ; Faizan and Lohmann 2018 ; Faizan et al. 2017 ; Kwankajornkiet et al. 2016 ; Susanti et al. 2015 ) selected distractors that are declared in a KB to be siblings of the key, which also implies some notion of similarity (siblings are assumed to be similar). Another approach that relies on structured knowledge sources is described in Seyler et al. ( 2017 ). The authors used query relaxation, whereby queries used to generate question keys are relaxed to provide distractors that share some of the key features. Faizan and Lohmann ( 2018 ) and Faizan et al. ( 2017 ) and Stasaski and Hearst ( 2017 ) adopted a similar approach for selecting distractors. Others, including Liang et al. ( 2017 , 2018 ) and Liu et al. ( 2018 ), used ML-models to rank distractors based on a combination of the previous features.

Again, some distractor selection approaches are tailored to specific types of questions. For example, for pronoun reference questions generated in Satria and Tokunaga ( 2017a , 2017b ), words selected as distractors do not belong to the same coreference chain as this would make them correct answers. Another example of a domain specific approach for distractor selection is related to gap-fill questions. Kumar et al. ( 2015b ) ensured that distractors fit into the question sentence by calculating the probability of their occurring in the question.

Feedback generation: Feedback provides an explanation of the correctness or incorrectness of responses to questions, usually in reaction to user selection. As feedback generation is one of the main interests of this review, we elaborate more fully on this in the “ Feedback Generation ” section.

Controlling difficulty: This task focuses on determining how easy or difficult a question will be. We elaborate more on this in the section titled “ Difficulty ” .

Post-processing

The goal of post-processing is to improve the output questions. This is usually achieved via two processes:

Verbalisation: This task is concerned with producing the final surface structure of the question. There is more on this in the section titled “ Verbalisation ”.

Question ranking (also referred to as question selection or question filtering): Several generators employed an “over-generate and rank” approach whereby a large number of questions are generated, and then ranked or filtered in a subsequent phase. The ranking goal is to prioritise good quality questions. The ranking is achieved by the use of statistical models as in Blšták ( 2018 ), Kwankajornkiet et al. ( 2016 ), Liu et al. ( 2017 ), and Niraula and Rus ( 2015 ).

In this section, we summarise our observations on which input formats are most popular in the literature published after 2014. One question we had in mind is whether structured sources (i.e. whereby knowledge is organised in a way that facilitates automatic retrieval and processing) are gaining more popularity. We were also interested in the association between the input being used and the domain or question types. Specifically, are some inputs more common in specific domains? And are some inputs more suitable for specific types of questions?

As in the findings of Alsubait (Table  1 ), text is still the most popular type of input with 42 studies using it. Ontologies and resource description framework (RDF) knowledge bases come second, with eight and six studies, respectively, using these. Note that these three input formats are shared between our review and Alsubit’s review. Another input, used by more than one study, are question stems and keys, which feature in five studies that focus on generating distractors. See the Appendix “ Summary of Included Studies ” for types of inputs used in each study.

The majority of studies reporting the use of text as the main input are centred around generating questions for language learning (18 studies) or generating simple factual questions (16 studies). Other domains investigated are medicine, history, and sport (one study each). On the other hand, among studies utilising Semantic Web technologies, only one tackles the generation of language questions and nine tackle the generation of domain-unspecific questions. Questions for biology, medicine, biomedicine, and programming have also been generated using Semantic Web technologies. Additional domains investigated in Alsubait’s review are mathematics, science, and databases (for studies using the Semantic Web). Combining both results, we see a greater variety of domains in semantic-based approaches.

Free-response questions are more prevalent among studies using text, with 21 studies focusing on this question type, 18 on multiple-choice, three on both free-response and multiple-choice questions, and one on verbal response questions. Some studies employ additional resources such as WordNet (Kwankajornkiet et al. 2016 ; Kumar et al. 2015a ) or DBpedia (Faizan and Lohmann 2018 ; Faizan et al. 2017 ; Tamura et al. 2015 ) to generate distractors. By contrast, MCQs are more prevalent in studies using Semantic Web technologies, with ten studies focusing on the generation of multiple-choice questions and four studies focusing on free-response questions. This result is similar to those obtained by Alsubait (Table  1 ) with free-response being more popular for generation from text and multiple-choice more popular from structured sources. We have discussed why this is the case in the “ Findings of Alsubait’s Review ” Section.

Domain, Question Types and Language

As Alsubait found previously (“ Findings of Alsubait’s Review ” section), language learning is the most frequently investigated domain. Questions generated for language learning target reading comprehension skills, as well as knowledge of vocabulary and grammar. Research is ongoing concerning the domains of science (biology and physics), history, medicine, mathematics, computer science, and geometry, but there are still a small number of papers published on these domains. In the current review, no study has investigated the generation of logic and analytical reasoning questions, which were present in the studies included in Alsubait’s review. Sport is the only new domain investigated in the reviewed literature. Table  5 shows the number of papers in each domain and the types of questions generated for these domains (for more details, see the Appendix, “ Summary of Included Studies ”). As Table  5 illustrates, gap-fill and wh-questions are again the most popular. The reader is referred to the section “ Findings of Alsubait’s Review ” for our discussion of reasons for the popularity of the language domain and the aforementioned question types.

With regard to the response format of questions, both free- and selected-response questions (i.e. MC and T/F questions) are of interest. In all, 35 studies focus on generating selected-response questions, 32 on generating free-response questions, and four studies on both. These numbers are similar to the results reported in Alsubait ( 2015 ), which were 33 and 32 papers on generation of free- and selected-response questions respectively (Table  1 ). However, which format is more suitable for assessment is debatable. Although some studies that advocate the use of free-response argue that these questions can test a higher cognitive level, Footnote 10 most automatically generated free-response questions are simple factual questions for which the answers are short facts explicitly mentioned in the input. Thus, we believe that it is useful to generate distractors, leaving to exam designers the choice of whether to use the free-response or the multiple-choice version of the question.

Concerning language, the majority of studies focus on generating questions in English (59 studies). Questions in Chinese (5 studies), Japanese (3 studies), Indonesian (2 studies), as well as Punjabi and Thai (1 study each) have also been generated. To ascertain which languages have been investigated before, we skimmed the papers identified in Alsubait ( 2015 ) and found three studies on generating questions in languages other than English: French in Fairon ( 1999 ), Tagalog in Montenegro et al. ( 2012 ), and Chinese, in addition to English, in Wang et al. ( 2012 ). This reflects an increasing interest in generating questions in other languages, which possibly accompanies interest in NLP research in these domains. Note that there may be studies on other languages or more studies on the languages we have identified that we were not able to capture, because we excluded studies written in languages other than English.

Feedback Generation

Feedback generation concerns the provision of information regarding the response to a question. Feedback is important in reinforcing the benefits of questions especially in electronic environments in which interaction between instructors and students is limited. In addition to informing test takers of the correctness of their responses, feedback plays a role in correcting test takers’ errors and misconceptions and in guiding them to the knowledge they must acquire, possibly with reference to additional materials.

This aspect of questions has been neglected in early and recent AQG literature. Among the literature that we reviewed, only one study, Leo et al. ( 2019 ), has generated feedback, alongside the generated questions. They generate feedback as a verbalisation of the axioms used to select options. In cases of distractors, axioms used to generate both key and distractors are included in the feedback.

We found another study (Das and Majumder 2017 ) that has incorporated a procedure for generating hints using syntactic features, such as the number of words in the key, the first two letters of a one-word key, or the second word of a two-words key.

Difficulty is a fundamental property of questions that is approximated using different statistical measures, one of which is percentage correct (i.e the percentage of examinees who answered a question correctly). Footnote 11 Lack of control over difficulty poses issues such as generating questions of inappropriate difficulty (inappropriately easy or difficult questions). Also, searching for a question with a specific difficulty among a huge number of generated questions is likely to be tedious for exam designers.

We structure this section around three aspects of difficulty models: 1) their generality, 2) features underlying them, and 3) evaluation of their performance.

Despite the growth in AQG, only 14 studies have dealt with difficulty. Eight of these studies focus on the difficulty of questions belonging to a particular domain, such as mathematical word problems (Wang and Su 2016 ; Khodeir et al. 2018 ), geometry questions (Singhal et al. 2016 ), vocabulary questions (Susanti et al. 2017a ), reading comprehension questions (Gao et al. 2018 ), DFA problems (Shenoy et al. 2016 ), code-tracing questions (Thomas et al. 2019 ), and medical case-based questions (Leo et al. 2019 ; Kurdi et al. 2019 ). The remaining six focus on controlling the difficulty of non-domain-specific questions (Lin et al. 2015 ; Alsubait et al. 2016 ; Kurdi et al. 2017 ; Faizan and Lohmann 2018 ; Faizan et al. 2017 ; Seyler et al. 2017 ; Vinu and Kumar 2015a , 2017a ; Vinu et al. 2016 ; Vinu and Kumar 2017b , 2015b ).

Table  6 shows the different features proposed for controlling question difficulty in the aforementioned studies. In seven studies, RDF knowledge bases or OWL ontologies were used to derive the proposed features. We observe that only a few studies account for the contribution of both stem and options to difficulty.

Difficulty control was validated by checking agreement between predicted difficulty and expert prediction in Vinu and Kumar ( 2015b ), Alsubait et al. ( 2016 ), Seyler et al. ( 2017 ), Khodeir et al. ( 2018 ), and Leo et al. ( 2019 ), by checking agreement between predicted difficulty and student performance in Alsubait et al. ( 2016 ), Susanti et al. ( 2017a ), Lin et al. ( 2015 ), Wang and Su ( 2016 ), Leo et al. ( 2019 ), and Thomas et al. ( 2019 ), by employing automatic solvers in Gao et al. ( 2018 ), or by asking experts to complete a survey after using the tool (Singhal et al. 2016 ). Expert reviews and mock exams are equally represented (seven studies each). We observe that the question samples used were small, with the majority of samples containing less than 100 questions (Table  7 ).

In addition to controlling difficulty, in one study (Kusuma and Alhamri 2018 ), the author claims to generate questions targeting a specific Bloom level. However, no evaluation of whether generated questions are indeed at a particular Bloom level was conducted.

Verbalisation

We define verbalisation as any process carried out to improve the surface structure of questions (grammaticality and fluency) or to provide variations of questions (i.e. paraphrasing). The former is important since linguistic issues may affect the quality of generated questions. For example, grammatical inconsistency between the stem and incorrect options enables test takers to select the correct option with no mastery of the required knowledge. On the other hand, grammatical inconsistency between the stem and the correct option can confuse test takers who have the required knowledge and would have been likely to select the key otherwise. Providing different phrasing for the question text is also of importance, playing a role in keeping test takers engaged. It also plays a role in challenging test takers and ensuring that they have mastered the required knowledge, especially in the language learning domain. To illustrate, consider questions for reading comprehension assessment; if the questions match the text with a very slight variation, test takers are likely to be able to answer these questions by matching the surface structure without really grasping the meaning of the text.

From the literature identified in this review, only ten studies apply additional processes for verbalisation. Given that the majority of the literature focuses on gap-fill question generation, this result is expected. Aspects of verbalisation that have been considered are pronoun substitutions (i.e. replacing pronouns by their antecedents) (Huang and He 2016 ), selection of a suitable auxiliary verb (Mazidi and Nielsen 2015 ), determiner selection (Zhang and VanLehn 2016 ), and representation of semantic entities (Vinu and Kumar 2015b ; Seyler et al. 2017 ) (see below for more on this). Other verbalisation processes that are mostly specific to some question types are the following: selection of singular personal pronouns (Faizan and Lohmann 2018 ; Faizan et al. 2017 ), which is relevant for Jeopardy questions; selection of adjectives for predicates (Vinu and Kumar 2017a ), which is relevant for aggregation questions; and ordering sentences and reference resolution (Huang and He 2016 ), which is relevant for word problems.

For approaches utilising structured knowledge sources, semantic entities, which are usually represented following some convention such as using camel case (e.g anExampleOfCamelCase) or using underscore as a word separator, need to be represented in a natural form. Basic processing which includes word segmentation, adaptation of camel case, underscores, spaces, punctuation, and conversion of the segmented phrase into a suitable morphological form (e.g. “has pet” to “having pet”), has been reported in Vinu and Kumar ( 2015b ). Seyler et al. ( 2017 ) used Wikipedia to verbalise entities, an entity-annotated corpus to verbalise predicates, and WordNet to verbalise semantic types. The surface form of Wikipedia links was used as verbalisation for entities. The annotated corpus was used to collect all sentences that contain mentions of entities in a triple, combined with some heuristic for filtering and scoring sentences. Phrases between the two entities were used as verbalisation of predicates. Finally, as types correspond to WordNet synsets, the authors used a lexicon that comes with WordNet for verbalising semantic types.

Only two studies (Huang and He 2016 ; Ai et al. 2015 ) have considered paraphrasing. Ai et al. ( 2015 ) employed a manually created library that includes different ways to express particular semantic relations for this purpose. For instance, “wife had a kid from husband” is expressed as “from husband, wife had a kid”. The latter is randomly chosen from among the ways to express the marriage relation as defined in the library. The other study that tackles paraphrasing is Huang and He ( 2016 ) in which words were replaced with synonyms.

In this section, we report on standard datasets and evaluation practices that are currently used in the field (considering how QG approaches are evaluated and what aspects of questions such evaluation focuses on). We also report on issues hindering comparison of the performance of different approaches and identification of the best-performing methods. Note that our focus is on the results of evaluating the whole generation approach, as indicated by the quality of generated questions, and not on the results of evaluating a specific component of the approach (e.g. sentence selection or classification of question types). We also do not report on evaluations related to the usability of question generators (e.g. evaluating ease of use) or efficiency (i.e. time taken to generate questions). For approaches using ontologies as the main input, we consider whether they use existing ontologies or experimental ones (i.e. created for the purpose of QG), since Alsubait ( 2015 ) has concerns related to using experimental ontologies in evaluations (see “ Findings of Alsubait’s Review ” section). We also reflect on further issues in the design and implementation of evaluation procedures and how they can be improved.

Standard Datasets

In what follows, we outline publicly available question corpora, providing details about their content, as well as how they were developed and used in the context of QG. These corpora are grouped on the basis of the initial purpose for which they were developed. Following this, we discuss the advantages and limitations of using such datasets and call attention to some aspects to consider when developing similar datasets.

The identified corpora are developed for the following three purposes:

Machine reading comprehension

The Stanford Question Answering Dataset (SQuAD) Footnote 12 (Rajpurkar et al. 2016 ) consists of 150K questions about Wikipedia articles developed by AMT co-workers. Of those, 100K questions are accompanied by paragraph-answer pairs from the same articles and 50K questions have no answer in the article. This dataset was used by Kumar et al. ( 2018 ) and Wang et al. ( 2018 ) to perform a comparison among variants of the generation approach they developed and between their approach and an approach from the literature. The comparison was based on the metrics BLEU-4, METEOR, and ROUGE-L which capture the similarity between generated questions and the SQuAD questions that serve as ground truth questions (there is more information on these metrics in the next section). That is, questions were generated using the 100K paragraph-answer pairs as input. Then, the generated questions were compared with the human-authored questions that are based on the same paragraph-answer pairs.

NewsQA Footnote 13 is another crowd-sourced dataset of about 120K question-answer pairs about CNN articles. The dataset consists of wh-questions and is used in the same way as SQuAD.

Training question-answering (QA) systems

The 30M factoid question-answer corpus (Serban et al. 2016 ) is a corpus of questions automatically generated from Freebase. Footnote 14 Freebase triples (of the form: subject, relationship, object) were used to generate questions where the correct answer is the object of the triple. For example, the question: “What continent is bayuvi dupki in?” is generated from the triple (bayuvi dupki, contained by, europe). The triples and the questions generated from them are provided in the dataset. A sample of the questions was evaluated by 63 AMT co-workers, each of whom evaluated 44-75 examples; each question was evaluated by 3-5 co-workers. The questions were also evaluated by automatic evaluation metrics. Song and Zhao ( 2016a ) performed a qualitative analysis comparing the grammaticality and naturalness of questions generated by their approach and questions from this corpus (although the comparison is not clear).

SciQ Footnote 15 (Welbl et al. 2017 ) is a corpus of 13.7K science MCQs on biology, chemistry, earth science, and physics. The questions target a broad cohort, ranging from elementary to college introductory level. The corpus was created by AMT co-workers at a cost of $10,415 and its development relied on a two-stage procedure. First, 175 co-workers were shown paragraphs and asked to generate questions for a payment of $0.30 per question. Second, another crowd-sourcing task in which co-workers validate the questions developed and provide them with distractors was conducted. A list of six distractors was provided by a ML-model. The co-workers were asked to select two distractors from the list and to provide at least one additional distractor for a payment of $0.20. For evaluation, a third crowd-sourcing task was created. The co-workers were provided with 100 question pairs, each pair consisting of an original science exam question and a crowd-sourced question in a random order. They were instructed to select the question likelier to be the real exam question. The science exam questions were identified in 55% of the cases. This corpus was used by Liang et al. ( 2018 ) to develop and test a model for ranking distractors. All keys and distractors in the dataset were fed to the model to rank. The authors assessed whether ranked distractors were among the original distractors provided with the questions.

Question generation

The question generation shared task challenge (QGSTEC) dataset Footnote 16 (Rus et al. 2012 ) is created for the QG shared task. The shared task contains two challenges: question generation from individual sentences and question generation from a paragraph. The dataset contains 90 sentences and 65 paragraphs collected from Wikipedia, OpenLearn, Footnote 17 and Yahoo! Answers, with 180 and 390 questions generated from the sentences and paragraphs, respectively. A detailed description of the dataset, along with the results achieved by the participants, is given in Rus et al. ( 2012 ). Blšták and Rozinajová ( 2017 , 2018 ) used this dataset to generate questions and compare their performance on correctness to the performance of the systems participating in the shared task.

Medical CBQ corpus (Leo et al. 2019 ) is a corpus of 435 case-based, auto-generated questions that follow four templates (“What is the most likely diagnosis?”, “What is the drug of choice?”, “What is the most likely clinical finding?”, and “What is the differential diagnosis?”). The questions are accompanied by experts’ ratings of appropriateness, difficulty, and actual student performance. The data was used to evaluate an ontology-based approach for generating case-based questions and predicting their difficulty.

MCQL is a corpus of about 7.1K MCQs crawled from the web, with an average of 2.91 distractors per question. The domains of the questions are biology, physics, and chemistry, and they target Cambridge O-level and college-level. The dataset was used in Blšták and Rozinajová ( 2017 ) to develop and evaluate a ML-model for ranking distractors.

Several datasets were used for assessing the ability of question generators to generate similar questions (see Table  8 for an overview). Note that the majority of these datasets were developed for purposes other than education and, as such, the educational value of the questions has not been validated. Therefore, while use of these datasets supports the claim of being able to generate human-like questions, it does not indicate that the generated questions are good or educationally useful. Additionally, restricting the evaluation of generation approaches to the criterion of being able to generate questions that are similar to those in the datasets does not capture their ability to generate other good quality questions that differ in surface structure and semantics.

Some of these datasets were used to develop and evaluate ML-models for ranking distractors. However, being written by humans does not necessarily mean that these distractors are good. This is, in fact, supported by many studies on the quality of distractors in real exam questions (Sarin et al. 1998 ; Tarrant et al. 2009 ; Ware and Vik 2009 ). If these datasets were to be used for similar purposes, distractors would need to be filtered based on their functionality (i.e. being picked by test takers as answers to questions).

We also observe that these datasets have been used in a small number of studies (1-2). This is partially due to the fact that many of them are relatively new. In addition, the design space for question generation is large (i.e. different inputs, question types, and domains). Therefore, each of these datasets is only relevant for a small set of question generators.

Types of Evaluation

The most common evaluation approach is expert-based evaluation (n = 21), in which experts are presented with a sample of generated questions to review. Given that expert review is also a standard procedure for selecting questions for real exams, expert rating is believed to be a good proxy for quality. However, it is important to note that expert review only provides initial evidence for the quality of questions. The questions also need to be administered to a sample of students to obtain further evidence of their quality (empirical difficulty, discrimination, and reliability), as we will see later. However, invalid questions must be filtered first, and expert review is also utilised for this purpose, whereby questions indicated by experts to be invalid (e.g. ambiguous, guessable, or not requiring domain knowledge) are filtered out. Having an appropriate question set is important to keep participants involved in question evaluation motivated and interested in solving these questions.

One of our observations on expert-based evaluation is that only in a few studies were experts required to answer the questions as part of the review. We believe this is an important step to incorporate since answering a question encourages engagement and triggers deeper thinking about what is required to answer. In addition, expert performance on questions is another indicator of question quality and difficulty. Questions answered incorrectly by experts can be ambiguous or very difficult.

Another observation on expert-based evaluation is the ambiguity of instructions provided to experts. For example, in an evaluation of reading comprehension questions (Mostow et al. 2017 ), the authors reported different interpretations of the instructions for rating the overall question quality, whereby one expert pointed out that it is not clear whether reading the preceding text is required in order to rate the question as being of good quality. Researchers have also measured question acceptability, as well as other aspects of questions, using scales with a large number of categories (up to a 9-point scale) without a clear categorisation for each category. Zhang ( 2015 ) found that reviewers perceive scale differently and not all categories of scales are used by all reviewers. We believe that these two issues are reasons for low inter-rater agreement between experts. To improve the accuracy of the data obtained through expert review, researchers must precisely specify the criteria by which to evaluate questions. In addition, a pilot test needs to be conducted with experts to provide an opportunity for validating the instructions and ensuring that instructions and questions are easily understood and interpreted as intended by different respondents.

The second most commonly employed method for evaluation is comparing machine-generated questions (or parts of questions) to human-authored ones (n = 15), which is carried out automatically or as part of the expert review. This comparison is utilised to confirm different aspects of question quality. Zhang and VanLehn ( 2016 ) evaluated their approach by counting the number of questions in common between those that are human- and machine-generated. The authors used this method under the assumption that humans are likely to ask deep questions about topics (i.e. questions of higher cognitive level). On this ground, the authors claimed that an overlap means the machine was able to mimic this in-depth questioning. Other researchers have compared machine-generated questions with human-authored reference questions using metrics borrowed from the fields of text summarisation (ROUGE (Lin 2004 )) and machine translation (BLEU (Papineni et al. 2002 ) and METEOR (Banerjee and Lavie 2005 )). These metrics measure the similarity between two questions generated from the same text segment or sentence. Put simply, this is achieved by counting matching n-grams in the gold-standard question to n-grams in the generated question with some focusing on recall (i.e. how much of the reference question is captured in the generated question) and others focusing on precision (i.e. how much of the generated question is relevant). METEOR also considers stemming and synonymy matching. Wang et al. ( 2018 ) claimed that these metrics can be used as initial, inexpensive, large-scale indicators of the fluency and relevancy of questions. Other researchers investigated whether machine-generated questions are indistinguishable from human-authored questions by mixing both types and asking experts about the source of each question (Chinkina and Meurers 2017 ; Susanti et al. 2015 ; Khodeir et al. 2018 ). Some researchers evaluated their approaches by investigating the ability of the approach to assemble human-authored distractors. For example, Yaneva et al. ( 2018 ) only focused on generating distractors given a question stem and key. However, given the published evidence of the poor quality of human-generated distractors, additional checks need to be performed, such as the functionality of these distractors.

Crowd-sourcing has also been used in ten of the studies. In eight of these, co-workers were employed to review questions while in the remaining three, they were employed to take mock tests. To assess the quality of their responses, Chinkina et al. ( 2017 ) included test questions to make sure that the co-workers understood the task and were able to distinguish low-quality from high-quality questions. However, including a process for validating the reliability of co-workers has been neglected in most studies (or perhaps not reported). Another validation step that can be added to the experimental protocol is conducting a pilot to test the capability of co-workers for review. This can also be achieved by adding validated questions to the list of questions to be reviewed by the co-workers (given the availability of a validated question set).

Similarly, students have been employed to review questions in nine studies and to take tests in a further ten. We attribute the low rate of question validation through testing with student cohorts to it being time-consuming and to the ethical issues involved in these experiments. Experimenters must ensure that these tests do not have an influence on students’ grades or motivations. For example, if multiple auto-generated questions focus on one topic, students could perceive this as an important topic and pay more attention to it while studying for upcoming exams, possibly giving less attention to other topics not covered by the experimental exam. Difficulty of such experimental exams could also affect students. If an experimental test is very easy, students could expect upcoming exams to be the same, again paying less attention when studying for them. Another possible threat is a drop in student motivation triggered by an experimental exam being too difficult.

Finally, for ontology-based approaches, similar to the findings reported in the section “ Findings of Alsubait’s Review ”, most ontologies used in evaluations were hand-crafted for experimental purposes and the use of real ontologies was neglected, except in Vinu and Kumar ( 2015b ), Leo et al. ( 2019 ), and Lopetegui et al. ( 2015 ).

Quality Criteria and Metrics

Table  9 shows the criteria used for evaluating the quality of questions or their components. Some of these criteria concern the linguistic quality of questions, such as grammatical correctness, fluency, semantic ambiguity, freeness from errors, and distractor readability. Others are educationally oriented, such as educational usefulness, domain relevance, and learning outcome. There are also standard quality metrics for assessing questions, such as difficulty, discrimination, and cognitive level. Most of the criteria can be used to evaluate any type of question and only a few are applicable to a specific class of questions, such as the quality of blank (i.e. a word or a phrase that is removed from a segment of text) in gap-fill questions. As can be seen, human-based measures are the most common compared to automatic scoring and statistical procedures. More details about the measurement of these criteria and the results achieved by generation approaches can be found in the Appendix “ Evaluation ”.

Performance of Generation Approaches and Gold Standard Performance

We started this systematic review hoping to identify standard performance and the best generation approaches. However, a comparison between the performances of various approaches was not possible due to heterogeneity in the measurement of quality and reporting of results. For example, scales that consist of different number of categories were used by different studies for measuring the same variables. We were not able to normalise these scales because most studies have only reported aggregated data without providing the number of observations in each rating scale category. Another example of heterogeneity is difficulty based on examinee performance. While some studies use percentage correct, others use Rasch difficulty without providing the raw data to allow the other metric to be calculated. Also, essential information that is needed to judge the trustability and generality of the results, such as sample size and selection method, was not reported in multiple studies. All of these issues preclude a statistical analysis of, and a conclusion about, the performance of generation approaches.

Quality Assessment Results

In this section, we describe and reflect on the state of experimental reporting in the reviewed literature.

Overall, the experimental reporting is unsatisfactory. Essential information that is needed to assess the strength of a study is not reported, raising concerns about trustability and generalisability of the results. For example, the number of evaluated questions, the number of participants involved in evaluations, or both of these numbers are not mentioned in five, ten and five studies, respectively. Information about sampling strategy and how sample size was determined is almost never reported (see the Appendix, “ Quality assessment ”).

A description of the participants’ characteristics, whether experts, students, or co-workers, is frequently missing (neglected by 23 studies). Minimal information that needs to be reported about experts involved in reviewing questions, in addition to their numbers, is their teaching and exam construction experience. Reporting whether experts were paid or not is important for the reader to understand possible biases involved. However, this is not reported in 51 studies involving experiments with human subjects. Other additional helpful information to report is the time taken to review, because this would assist researchers to estimate the number of experts to recruit given a particular sample size, or to estimate the number of questions to sample given the available number of experts.

Characteristics of students involved in evaluations, such as their educational level and experience with the subject under assessment, are important for replication of studies. In addition, this information can provide a basis for combining evidence from multiple studies. For example, we could gain stronger evidence about the effect of specific features on question difficulty by combining studies investigating the same features with different cohorts. In addition, the characteristics of the participants are a possible justification for the difference in difficulty between studies. Similarly, criteria used for the selection of co-workers such as imposing a restriction on which countries they are from, or the number and accuracy of previous tasks in which they participated is important.

Some studies neglect to report on the total number of generated questions and the distribution of questions per categories (question types, difficulty levels, and question sources, when applicable), which are necessary to assess the suitability of sampling strategies. For example, without reporting the distribution of question types, making a claim based on random sampling that “70% of questions are appropriate to be used in exams” would be misleading if the distribution of question types is skewed. This is due to the sample not being representative of question types with a low number of questions. Similarly, if the majority of generated questions are easy, using a random sample will result in the underrepresentation of difficult questions, consequently precluding any conclusion about difficult questions or any comparison between easy and difficult questions.

With regard to measurement descriptions, 10 studies fail to report information sufficient for replication, such as instructions given to participants and a description of the rating scales. Another limitation concerning measurements is the lack of assessment of inter-rater reliability (not reported by 43 studies). In addition, we observed a lack of justification for experimental decisions. Examples of this are the sources from which questions were generated, when particular texts or knowledge sources were selected without any discussion of whether these sources were representative and of what they were representative. We believe that generation challenges and question quality issues that might be encountered when using different sources need to be raised and discussed.

Conclusion and Future Work

In this paper, we have conducted a comprehensive review of 93 papers addressing the automatic generation of questions for educational purposes. In what follows, we summarise our findings in relation to the review objectives.

Providing an Overview of the AQG Community and its Activities

We found that AQG is an increasing activity of a growing community. Through this review, we identified the top publication venues and the active research groups in the field, providing a connection point for researchers interested in the field.

Summarising Current QG Approaches

We found that the majority of QG systems focus on generating questions for the purpose of assessment. The template-based approach was the most common method employed in the reviewed literature. In addition to the generation of complete questions or of question components, a variety of pre- and post-processing tasks that are believed to improve question quality have been investigated. The focus was on the generation of questions from text and for the language domain. The generation of both multiple-choice and free-response questions was almost equally investigated with a large number of studies focusing on wh-word and gap-fill questions. We also found increased interest in generating questions in languages other than English. Although extensive research has been carried out on QG, only a small proportion of these tackle the generation of feedback, verbalisation of questions, and the control of question difficulty.

Identifying Gold Standard performance in AQG

Incomparability of the performance of generation approaches is an issue we identified in the reviewed literature. This issue is due to the heterogeneity in both measurement of quality and reporting of results. We suggest below how the evaluation of questions and reporting of results can be improved to overcome this issue.

Tracking the Evolution of AQG Since Alsubait’s Review

Our results are consistent with the findings of Alsubait ( 2015 ). Based on these findings, we suggest that research in the area can be extended in the following directions (starting at the question level before moving on to the evaluation and research in closely related areas):

Improvement at the Question Level

Generating questions with controlled difficulty.

As mentioned earlier, there is little research on question difficulty and what there is mostly focuses on either stem or distractor difficulty. The difficulty of both stem and options plays a role in overall difficulty and therefore needs to be considered together and not in isolation. Furthermore, controlling MCQ difficulty by varying the similarity between key and distractors is a common feature found in multiple studies. However, similarity is only one facet of difficulty and there are others that need to be identified and integrated into the generation process. Thus, the formulation of a theory behind an intelligent automatic question generator capable of both generating questions and accurately controlling their difficulty is at the heart of AQG research. This would be used for improving the quality of generated questions by filtering inappropriately easy or difficult questions which is especially important given the large number of questions.

Enriching Question Forms and Structures

One of the main limitations of existing works is the simplicity of generated questions, which has also been highlighted in Song and Zhao ( 2016b ). Most generated questions consist of a few terms and target lower cognitive levels. While these questions are still useful, there is a potential for improvement by exploring the generation of other, higher order and more complex, types of questions.

Automating Template Construction

The template library is a major component of question generation systems. At present, the process of template construction is largely manual. The templates are either developed through analysing a set of hand-written questions manually or through consultation with domain experts. While one of the main motivations for generating questions automatically is cost reduction, both of these template acquisition techniques are costly. In addition, there is no evidence that the set of templates defined by a few experts is typical of the set of questions used in assessments. We attribute part of the simplicity of the current questions to the cost, both in terms of time and resources, of both template acquisition techniques.

The cost of generating questions automatically could be reduced further by automatically constructing templates. In addition, this would contribute to the development of more diverse questions.

Employing natural language generation and processing techniques in order to present questions in natural and correct forms and to eliminate errors that invalidate questions, such as syntactic clues, are important steps to take before questions can be used beyond experimental settings for assessment purposes.

As has been seen in both reviews, work on feedback generation is almost non-existent. Developing mechanisms for producing rich, effective feedback is one of the features that needs to be integrated into the generation process. This includes different types of feedback, such as formative, summative, interactive, and personalised feedback.

Improvement of Evaluation Methods

Using human-authored questions for evaluation.

Evaluating question quality, whether by means of expert review or mock exams, is an expensive and time consuming process. Analysing existing exam performance data is a potential source for evaluating question quality and difficulty prediction models. Translating human-authored questions to a machine-processable representation is a possible method for evaluating the ability of generation approaches to generate human-like questions. Regarding the evaluation of difficulty models, this can be done by translating questions to a machine-processable representation, computing the features of these questions, and examining their effect on difficulty. This analysis also provides an understanding of pedagogical content knowledge (i.e. concepts that students often find difficult and usually have misconceptions about). This knowledge can be integrated into difficulty prediction models, or used for question selection and feedback generation.

Standardisation and Development of Automatic Scoring Procedures

To ease comparison between different generation approaches, which was difficult due to heterogeneity in measurement and reporting as well as ungrounded heterogeneity needs to be eliminated. The development of standard and well defined scoring procedures is important to reduce heterogeneity and improve inter-rater reliability. In addition, developing automatic scoring procedures that correlate with human ratings are also important since this will reduce evaluation cost and heterogeneity.

Improvement of Reporting

We also emphasise the need for good experimental reporting. In general, authors should improve reporting on their generation approaches and on evaluation, which are both essential for other researchers who wish to compare their approaches with existing approaches. At a minimum, data extracted in this review (refer to questions under OBJ2 and OBJ3) should be reported in all publications on AQG. To ensure quality, journals can require authors to be complete a checklist prior to peer review, which has shown to improve the reporting quality (Han et al. 2017 ). Alternatively, text-mining techniques can be used for assessing the reporting quality by targeting key information in AQG literature, as has been proposed in Flórez-Vargas et al. ( 2016 ).

Other Areas of Improvement and Further Research

Assembling exams from the generated questions.

Although there is a large amount of work that needs to be done at the question level before moving to the exam level, further work in extending the difficulty models, enriching question form and structure, and improving presentation are steps towards this goal. Research in these directions will open new opportunities for AQG research to move towards assembling exams automatically from generated questions. One of the challenges in exam generation is the selection of a question set that is of appropriate difficulty with good coverage of the material. Ensuring that questions do not overlap or provide clues for other questions also needs to be taken into account. The AQG field could adopt ideas from the question answering field in which question entailment has been investigated (for example, see the work of Abacha and Demner-Fushman ( 2016 )). Finally, ordering questions in a way that increases motivation and maximises the accuracy of scores is another interesting area.

Mining Human-Authored Questions

While existing researchers claim that the questions they generate can be used for educational purposes, these claims are not generally supported. More attention needs to be given to the educational value of generated questions.

In addition to potential use in evaluation, analysing real, good quality exams can help to gain insights into what questions need to be generated so that the generation addresses real life educational needs. This will also help to quantify the characteristics of real questions (e.g. number of terms in real questions) and direct attention to what needs to be done and where the focus should be in order to move to exam generation. Additionally, exam questions reflect what should be included in similar assessments that, in turn, can be further used for content selection and the ranking of questions. For example, concepts extracted from these questions can inform the selection of existing textual or structured sources and the quantifying of whether or not the contents are of educational relevance.

Other potential advantages that the automatic mining of questions offers are the extraction of question templates, a major component of automatic question generators, and improving natural language generation. Besides, mapping the information contained in existing questions to an ontology permits modification of these questions, prediction of their difficulty, and the formation of theories about different aspects of the questions such as their quality.

Similarity Computation and Optimisation

A variety of similarity measures have been used in the context of QG to select content for questions, to select plausible distractors and to control question difficulty (see “ Generation Tasks ” section for examples). Similarity can also be employed in suggesting a diverse set of generated questions (i.e. questions that do not entail the same meaning regardless of their surface structure). Improving computation of the similarity measures (i.e. speed and accuracy) and investigating other types of similarity that might be needed for other question forms are all considered sidelines that have direct implications for improving the current automatic question generation process. Evaluating the performance of existing similarity measures in comparison to each other and whether or not cheap similarity measures can approximate expensive ones are further interesting objects of study.

Source Acquisition and Enrichment

As we have seen in this review, structured knowledge sources have been a popular source for question generation, either by themselves or to complement texts. However, knowledge sources are not available for many domains, while those that are developed for purposes other than QG might not be rich enough to generate good quality questions. Therefore, they need to be adapted or extended before they can be used for QG. As such, investigating different approaches for building or enriching structured knowledge sources and gaining further evidence for the feasibility of obtaining good quality knowledge sources that can be used for question generation, are crucial ingredients for their successful use in question generation.

Limitations

A limitation of this review is the underrepresentation of studies published in languages other than English. In addition, ten papers were excluded because of the unavailability of their full texts.

Questions like those presented in the T.V. show “Jeopardy!”. These questions consist of statements that give hints about the answer. See Faizan and Lohmann ( 2018 ) for an example.

Note that evaluated properties are not necessarily controlled by the generation method. For example, an evaluation could focus on difficulty and discrimination as an indication of quality.

The code and the input files are available at: https://github.com/grkurdi/AQG_systematic_review

The required sample size was calculated using the N.cohen.kappa function (Gamer et al. 2019 ).

This due to the initial description of Q9 being insufficient. However, the agreement improved after refining the description of Q9. demonstrating “moderate agreement”. Footnote 6 Note that Cohen’s kappa was unsuitable for assessing the agreement on the criteria Q6-Q8 due to the unbalanced distribution of responses (e.g. the majority of responses to Q6a were “no”). Since the level of agreement between both reviewers was high, the quality of the remaining studies was assessed by the first author.

Cohen’s kappa was interpreted according to the interpretation provided by Viera et al. ( 2005 ).

The last update of the search was on 3-4-2019.

http://www.questiongeneration.org/

Questions consisting of a text segment followed by a stem of the form: “The word X in paragraph Y is closest in meaning to:” and a set of options. See Susanti et al. ( 2015 ) for more details.

This relates to the processes required to answer questions as characterised in known taxonomies such as Bloom’s taxonomy (Bloom et al. 1956 ), SOLO taxonomy (Biggs and Collis 2014 ) or Webb’s depth of knowledge (Webb 1997 ).

A percentage of 0 means that no one answered the question correctly (highly difficult question), while 100% means that everyone answered the question correctly (extremely easy question).

This can be found at https://rajpurkar.github.io/SQuAD-explorer/

This can be found at https://datasets.maluuba.com/NewsQA

This is a collaboratively created knowledge base.

Available at http://allenai.org/data.html

The dataset can be obtained from https://github.com/bjwyse/QGSTEC2010/blob/master/QGSTEC-Sentences-2010.zip

OpenLearn is an online repository that provides access to learning materials from The Open University.

Abacha, AB, & Demner-Fushman, D. (2016). Recognizing question entailment for medical question answering. In: the AMIA annual symposium, American medical informatics association, p. 310.

Adithya, SSR, & Singh, PK. (2017). Web authoriser tool to build assessments using Wikipedia articles. In: TENCON 2017 - 2017 IEEE region 10 conference, pp. 467–470. https://doi.org/10.1109/TENCON.2017.8227909 .

Afzal, N. (2015). Automatic generation of multiple choice questions using surface-based semantic relations. International Journal of Computational Linguistics (IJCL) , 6 (3), 26–44. https://doi.org/10.1007/s00500-013-1141-4 .

Google Scholar  

Afzal, N, & Mitkov, R. (2014). Automatic generation of multiple choice questions using dependency-based semantic relations. Soft Computing , 18 (7), 1269–1281. https://doi.org/10.1007/s00500-013-1141-4 .

Article   Google Scholar  

Afzal, N, Mitkov, R, Farzindar, A. (2011). Unsupervised relation extraction using dependency trees for automatic generation of multiple-choice questions. In: Canadian conference on artificial intelligence, Springer, pp. 32–43. https://doi.org/10.1007/978-3-642-21043-3_4 .

Ai, R, Krause, S, Kasper, W, Xu, F, Uszkoreit, H. (2015). Semi-automatic generation of multiple-choice tests from mentions of semantic relations. In: the 2nd Workshop on Natural Language Processing Techniques for Educational Applications, pp. 26–33.

Alsubait, T. (2015). Ontology-based question generation . PhD thesis: University of Manchester.

Alsubait, T, Parsia, B, Sattler, U. (2012a). Automatic generation of analogy questions for student assessment: an ontology-based approach. Research in Learning Technology 20. https://doi.org/10.3402/rlt.v20i0.19198 .

Alsubait, T, Parsia, B, Sattler, U. (2012b). Mining ontologies for analogy questions: A similarity-based approach. In: OWLED.

Alsubait, T, Parsia, B, Sattler, U. (2012c). Next generation of e-assessment: automatic generation of questions. International Journal of Technology Enhanced Learning , 4 (3-4), 156–171.

Alsubait, T, Parsia, B, Sattler, U. (2013). A similarity-based theory of controlling MCQ difficulty. In 2013 2Nd international conference on e-learning and e-technologies in education (pp. 283–288). ICEEE: IEEE.. https://doi.org/10.1109/ICeLeTE.2013.664438

Alsubait, T, Parsia, B, Sattler, U. (2014a). Generating multiple choice questions from ontologies: Lessons learnt. In: OWLED, Citeseer, pp. 73–84.

Alsubait, T, Parsia, B, Sattler, U. (2014b). Generating multiple questions from ontologies: How far can we go? In: the 1st International Workshop on Educational Knowledge Management (EKM 2014), Linköping University Electronic Press, pp. 19–30.

Alsubait, T, Parsia, B, Sattler, U. (2016). Ontology-based multiple choice question generation. KI - Kü,nstliche Intelligenz , 30 (2), 183–188. https://doi.org/10.1007/s13218-015-0405-9 .

Araki, J, Rajagopal, D, Sankaranarayanan, S, Holm, S, Yamakawa, Y, Mitamura, T. (2016). Generating questions and multiple-choice answers using semantic analysis of texts. In The 26th international conference on computational linguistics (COLING , (Vol. 2016 pp. 1125–1136).

Banerjee, S, & Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In: the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72.

Basuki, S, & Kusuma, S F. (2018). Automatic question generation for 5w-1h open domain of Indonesian questions by using syntactical template-based features from academic textbooks. Journal of Theoretical and Applied Information Technology , 96 (12), 3908–3923.

Baturay, M H. (2015). An overview of the world of MOOCs. Procedia - Social and Behavioral Sciences , 174 , 427–433. https://doi.org/10.1016/j.sbspro.2015.01.685 .

Beck, JE, Mostow, J, Bey, J. (2004). Can automated questions scaffold children’s reading comprehension? In: International Conference on Intelligent Tutoring Systems, Springer, pp. 478–490.

Bednarik, L, & Kovacs, L. (2012a). Automated EA-type question generation from annotated texts, IEEE, SACI. https://doi.org/10.1109/SACI.2012.6250000 .

Bednarik, L, & Kovacs, L. (2012b). Implementation and assessment of the automatic question generation module, IEEE, CogInfoCom. https://doi.org/10.1109/CogInfoCom.2012.6421938 .

Biggs, J B, & Collis, KF. (2014). Evaluating the quality of learning: The SOLO taxonomy (Structure of the Observed Learning Outcome) . Cambridge: Academic Press.

Bloom, B S, Engelhart, M D, Furst, E J, Hill, W H, Krathwohl, D R. (1956). Taxonomy of educational objectives, handbook i: The cognitive domain vol 19 . New York: David McKay Co Inc.

Blšták, M. (2018). Automatic question generation based on sentence structure analysis. Information Sciences & Technologies: Bulletin of the ACM Slovakia , 10 (2), 1–5.

Blšták, M., & Rozinajová, V. (2017). Machine learning approach to the process of question generation. In Blšták, M., & Rozinajová, V. (Eds.) Text, speech, and dialogue (pp. 102–110). Cham: Springer International Publishing.. https://doi.org/10.1007/978-3-319-64206-2_12

Blšták, M., & Rozinajová, V. (2018). Building an agent for factual question generation task. In 2018 World symposium on digital intelligence for systems and machines (DISA) (pp. 143–150). IEEE.. https://doi.org/10.1109/DISA.2018.8490637

Bodenreider, O. (2004). The unified medical language system (UMLS): integrating biomedical terminology. Nucleic acids research , 32 (suppl_1), D267–D270. https://doi.org/10.1093/nar/gkh061 .

Boland, A, Cherry, M G, Dickson, R. (2013). Doing a systematic review: A student’s guide. Sage.

Ch, DR, & Saha, SK. (2018). Automatic multiple choice question generation from text: A survey. IEEE Transactions on Learning Technologies https://doi.org/10.1109/TLT.2018.2889100 , in press.

Chen, CY, Liou, HC, Chang, JS. (2006). Fast: an automatic generation system for grammar tests. In: the COLING/ACL on interactive presentation sessions, association for computational linguistics, pp. 1–4.

Chinkina, M, & Meurers, D. (2017). Question generation for language learning: From ensuring texts are read to supporting learning. In: the 12th workshop on innovative use of NLP for building educational applications, pp. 334–344.

Chinkina, M, Ruiz, S, Meurers, D. (2017). Automatically generating questions to support the acquisition of particle verbs: evaluating via crowdsourcing. In: CALL in a climate of change: adapting to turbulent global conditions, pp. 73–78.

Critical Appraisal Skills Programme. (2018). CASP qualitative checklist. https://casp-uk.net/wp-content/uploads/2018/03/CASP-Qualitative-Checklist-Download.pdf , accessed: 2018-09-07.

Das, B, & Majumder, M. (2017). Factual open cloze question generation for assessment of learner’s knowledge. International Journal of Educational Technology in Higher Education , 14 (1), 24. https://doi.org/10.1186/s41239-017-0060-3 .

Donnelly, K. (2006). SNOMED-CT: The Advanced terminology and coding system for eHealth. Studies in health technology and informatics , 121 , 279–290.

Downs, S H, & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. Journal of Epidemiology & Community Health , 52 (6), 377–384.

Fairon, C. (1999). A web-based system for automatic language skill assessment: Evaling. In: Symposium on computer mediated language assessment and evaluation in natural language processing, association for computational linguistics, pp. 62–67.

Faizan, A, & Lohmann, S. (2018). Automatic generation of multiple choice questions from slide content using linked data. In: the 8th International Conference on Web Intelligence, Mining and Semantics.

Faizan, A, Lohmann, S, Modi, V. (2017). Multiple choice question generation for slides. In: Computer Science Conference for University of Bonn Students, pp. 1–6.

Fattoh, I E, Aboutabl, A E, Haggag, M H. (2015). Semantic question generation using artificial immunity. International Journal of Modern Education and Computer Science , 7 (1), 1–8.

Flor, M, & Riordan, B. (2018). A semantic role-based approach to open-domain automatic question generation. In: the 13th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 254–263.

Flórez-Vargas, O., Brass, A, Karystianis, G, Bramhall, M, Stevens, R, Cruickshank, S, Nenadic, G. (2016). Bias in the reporting of sex and age in biomedical research on mouse models. eLife 5(e13615).

Gaebel, M, Kupriyanova, V, Morais, R, Colucci, E. (2014). E-learning in European higher education institutions: Results of a mapping survey conducted in october-December 2013 . Tech. rep.: European University Association.

Gamer, M, Lemon, J, Gamer, MM, Robinson, A, Kendall’s, W. (2019). Package ’irr’. https://cran.r-project.org/web/packages/irr/irr.pdf .

Gao, Y, Wang, J, Bing, L, King, I, Lyu. MR. (2018). Difficulty controllable question generation for reading comprehension. Tech. rep.

Goldbach, IR, & Hamza-Lup, FG. (2017). Survey on e-learning implementation in Eastern-Europe spotlight on Romania. In: the Ninth International Conference on Mobile, Hybrid, and On-Line Learning.

Gupta, M, Gantayat, N, Sindhgatta, R. (2017). Intelligent math tutor: Problem-based approach to create cognizance. In: the 4th ACM Conference on Learning@ Scale, ACM, pp. 241–244.

Han, S, Olonisakin, T F, Pribis, J P, Zupetic, J, Yoon, J H, Holleran, K M, Jeong, K, Shaikh, N, Rubio, D M, Lee, J S. (2017). A checklist is associated with increased quality of reporting preclinical biomedical research: a systematic review. PloS One , 12 (9), e0183591.

Hansen, J D, & Dexter, L. (1997). Quality multiple-choice test questions: Item-writing guidelines and an analysis of auditing testbanks. Journal of Education for Business , 73 (2), 94–97. https://doi.org/10.1080/08832329709601623 .

Heilman, M. (2011). Automatic factual question generation from text . PhD thesis: Carnegie Mellon University.

Heilman, M, & Smith, NA. (2009). Ranking automatically generated questions as a shared task. In: The 2nd Workshop on Question Generation, pp. 30–37.

Heilman, M, & Smith, NA. (2010a). Good question! statistical ranking for question generation. In: Human language technologies: The 2010 annual conference of the north american chapter of the association for computational linguistics, association for computational linguistics, pp. 609–617.

Heilman, M, & Smith, NA. (2010b). Rating computer-generated questions with mechanical turk. In: the NAACL HLT 2010 workshop on creating speech and language data with amazon’s mechanical turk, association for computational linguistics, pp. 35–40.

Hill, J, & Simha, R. (2016). Automatic generation of context-based fill-in-the-blank exercises using co-occurrence likelihoods and Google n-grams. In: the 11th workshop on innovative use of NLP for building educational applications, pp. 23–30.

Hingorjo, M R, & Jaleel, F. (2012). Analysis of one-best MCQs: the difficulty index, discrimination index and distractor efficiency. The Journal of the Pakistan Medical Association (JPMA) , 62 (2), 142–147.

Huang, Y, & He, L. (2016). Automatic generation of short answer questions for reading comprehension assessment. Natural Language Engineering , 22 (3), 457–489. https://doi.org/10.1017/S1351324915000455 .

Article   MathSciNet   Google Scholar  

Huang, Y T, & Mostow, J. (2015). Evaluating human and automated generation of distractors for diagnostic multiple-choice cloze questions to assess children’s reading comprehension. In Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (Eds.) Artificial intelligence in education (pp. 155–164). Cham: Springer International Publishing.

Huang, Y T, Tseng, Y M, Sun, Y S, Chen, MC. (2014). TEDQuiz: automatic quiz generation for TED talks video clips to assess listening comprehension. In 2014 IEEE 14Th international conference on advanced learning technologies (pp. 350–354). ICALT: IEEE.

Jiang, S, & Lee, J. (2017). Distractor generation for Chinese fill-in-the-blank items. In: the 12th workshop on innovative use of NLP for building educational applications, pp. 143–148.

Jouault, C, & Seta, K. (2014). Content-dependent question generation for history learning in semantic open learning space. In: The international conference on intelligent tutoring systems, Springer, pp. 300–305.

Jouault, C, Seta, K, Hayashi, Y. (2015a). A method for generating history questions using LOD and its evaluation. SIG-ALST of The Japanese Society for Artificial Intelligence , B5 (1), 28–33.

Jouault, C, Seta, K, Hayashi, Y. (2015b). Quality of LOD based semantically generated questions. In Conati, C., Heffernan, N., Mitrovic, A, Verdejo, M.F. (Eds.) Artificial intelligence in education (pp. 662–665). Cham: Springer International Publishing.

Jouault, C, Seta, K, Hayashi, Y. (2016a). Content-dependent question generation using LOD for history learning in open learning space. New Generation Computing , 34 (4), 367–394. https://doi.org/10.1007/s00354-016-0404-x .

Jouault, C, Seta, K, Yuki, H, et al. (2016b). Can LOD based question generation support work in a learning environment for history learning?. SIG-ALST , 5 (03), 37–41.

Jouault, C, Seta, K, Hayashi, Y. (2017). SOLS: An LOD based semantically enhanced open learning space supporting self-directed learning of history. IEICE Transactions on Information and Systems , 100 (10), 2556–2566.

Kaur, A, & Singh, S. (2017). Automatic question generation system for Punjabi. In: The international conference on recent innovations in science, Agriculture, Engineering and Management.

Kaur, J, & Bathla, A K. (2015). A review on automatic question generation system from a given Hindi text. International Journal of Research in Computer Applications and Robotics (IJRCAR) , 3 (6), 87–92.

Khodeir, N A, Elazhary, H, Wanas, N. (2018). Generating story problems via controlled parameters in a web-based intelligent tutoring system. The International Journal of Information and Learning Technology , 35 (3), 199–216.

Killawala, A, Khokhlov, I, Reznik, L. (2018). Computational intelligence framework for automatic quiz question generation. In: 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1–8. https://doi.org/10.1109/FUZZ-IEEE.2018.8491624 .

Kitchenham, B, & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering . Tech. rep.: Keele University and University of Durham.

Kovacs, L, & Szeman, G. (2013). Complexity-based generation of multi-choice tests in AQG systems, IEEE, CogInfoCom. https://doi.org/10.1109/CogInfoCom.2013.6719278 .

Kumar, G, Banchs, R, D’Haro, LF. (2015a). Revup: Automatic gap-fill question generation from educational texts. In: the 10th workshop on innovative use of NLP for building educational applications, pp. 154–161.

Kumar, G, Banchs, R, D’Haro, LF. (2015b). Automatic fill-the-blank question generator for student self-assessment. In: IEEE Frontiers in Education Conference (FIE), pp. 1–3. https://doi.org/10.1109/FIE.2015.7344291 .

Kumar, V, Boorla, K, Meena, Y, Ramakrishnan, G, Li, Y F. (2018). Automating reading comprehension by generating question and answer pairs. In Phung, D., Tseng, V.S., Webb, G.I., Ho, B., Ganji, M., Rashidi, L. (Eds.) Advances in knowledge discovery and data mining (pp. 335–348). Cham: Springer International Publishing.. https://doi.org/10.1007/978-3-319-93040-4_27

Kurdi, G, Parsia, B, Sattler, U. (2017). An experimental evaluation of automatically generated multiple choice questions from ontologies. In Dragoni, M., Poveda-Villalón, M., Jimenez-Ruiz, E. (Eds.) OWL: Experiences And directions – reasoner evaluation (pp. 24–39). Cham: Springer International Publishing.. https://doi.org/10.1007/978-3-319-54627-8_3

Kurdi, G, Leo, J, Matentzoglu, N, Parsia, B, Forege, S, Donato, G, Dowling, W. (2019). A comparative study of methods for a priori prediction of MCQ difficulty. the Semantic Web journal, In press.

Kusuma, S F, & Alhamri, R Z. (2018). Generating Indonesian question automatically based on Bloom’s taxonomy using template based method. KINETIK: Game Technology, Information System, Computer Network, Computing, Electronics, and Control , 3 (2), 145–152.

Kwankajornkiet, C, Suchato, A, Punyabukkana, P. (2016). Automatic multiple-choice question generation from Thai text. In: the 13th International Joint Conference on Computer Science and Software Engineering (JCSSE), pp. 1–6. https://doi.org/10.1109/JCSSE.2016.7748891 .

Le, N T, Kojiri, T, Pinkwart, N. (2014). Automatic question generation for educational applications – the state of art. In van Do, T., Thi, H.A.L, Nguyen, N.T. (Eds.) Advanced computational methods for knowledge engineering (pp. 325–338). Cham: Springer International Publishing.

Lee, CH, Chen, TY, Chen, LP, Yang, PC, Tsai, RTH. (2018). Automatic question generation from children’s stories for companion chatbot. In: 2018 IEEE International Conference on Information Reuse and Integration (IRI), pp. 491–494. https://doi.org/10.1109/IRI.2018.00078 .

Leo, J, Kurdi, G, Matentzoglu, N, Parsia, B, Forege, S, Donato, G, Dowling, W. (2019). Ontology-based generation of medical, multi-term MCQs. International Journal of Artificial Intelligence, in Education. https://doi.org/10.1007/s40593-018-00172-w .

Liang, C, Yang, X, Wham, D, Pursel, B, Passonneau, R, Giles, CL. (2017). Distractor generation with generative adversarial nets for automatically creating fill-in-the-blank questions. In: the Knowledge Capture Conference, p. 33. https://doi.org/10.1145/3148011.3154463 .

Liang, C, Yang, X, Dave, N, Wham, D, Pursel, B, Giles, CL. (2018). Distractor generation for multiple choice questions using learning to rank. In: the 13th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 284–290. https://doi.org/10.18653/v1/W18-0533 .

Lim, C S, Tang, K N, Kor, L K. (2012). Drill and practice in learning (and Beyond), Springer US, pp. 1040–1042. https://doi.org/10.1007/978-1-4419-1428-6_706 .

Lin, C, Liu, D, Pang, W, Apeh, E. (2015). Automatically predicting quiz difficulty level using similarity measures. In: the 8th International Conference on Knowledge Capture, ACM.

Lin, CY. (2004). ROUGE: A package for automatic evaluation of summaries. In: the Workshop on Text Summarization Branches Out.

Liu, M, & Calvo, RA. (2012). Using information extraction to generate trigger questions for academic writing support. In: the International Conference on Intelligent Tutoring Systems, Springer, pp. 358–367. https://doi.org/10.1007/978-3-642-30950-2_47 .

Liu, M, Calvo, RA, Aditomo, A, Pizzato, LA. (2012a). Using Wikipedia and conceptual graph structures to generate questions for academic writing support. IEEE Transactions on Learning Technologies , 5 (3), 251–263. https://doi.org/10.1109/TLT.2012.5 .

Liu, M, Calvo, RA, Rus, V. (2012b). G-Asks: An intelligent automatic question generation system for academic writing support. Dialogue & Discourse , 3 (2), 101–124. https://doi.org/10.5087/dad.2012.205 .

Liu, M, Calvo, R A, Rus, V. (2014). Automatic generation and ranking of questions for critical review. Journal of Educational Technology & Society , 17 (2), 333–346.

Liu, M, Rus, V, Liu, L. (2017). Automatic Chinese factual question generation. IEEE Transactions on Learning Technologies , 10 (2), 194–204. https://doi.org/10.1109/TLT.2016.2565477 .

Liu, M, Rus, V, Liu, L. (2018). Automatic Chinese multiple choice question generation using mixed similarity strategy. IEEE Transactions on Learning Technologies , 11 (2), 193–202. https://doi.org/10.1109/TLT.2017.2679009 .

Lopetegui, MA, Lara, BA, Yen, PY, Çatalyürek, Ü.V., Payne, PR. (2015). A novel multiple choice question generation strategy: alternative uses for controlled vocabulary thesauri in biomedical-sciences education. In: the AMIA annual symposium, american medical informatics association, pp. 861–869.

Majumder, M, & Saha, SK. (2015). A system for generating multiple choice questions: With a novel approach for sentence selection. In: the 2nd workshop on natural language processing techniques for educational applications, pp. 64–72.

Marrese-Taylor, E, Nakajima, A, Matsuo, Y, Yuichi, O. (2018). Learning to automatically generate fill-in-the-blank quizzes. In: the 5th workshop on natural language processing techniques for educational applications. https://doi.org/10.18653/v1/W18-3722 .

Mazidi, K. (2018). Automatic question generation from passages. In Gelbukh, A. (Ed.) Computational linguistics and intelligent text processing (pp. 655–665). Cham: Springer International Publishing.

Mazidi, K, & Nielsen, RD. (2014). Linguistic considerations in automatic question generation. In: the 52nd annual meeting of the association for computational linguistics, pp. 321–326.

Mazidi, K, & Nielsen, R D. (2015). Leveraging multiple views of text for automatic question generation. In Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (Eds.) Artificial intelligence in education (pp. 257–266). Cham: Springer International Publishing.

Mazidi, K, & Tarau, P. (2016a). Automatic question generation: From NLU to NLG Micarelli, A., Stamper, J., Panourgia K. (Eds.), Springer International Publishing, Cham. https://doi.org/10.1007/978-3-319-39583-8_3 .

Mazidi, K, & Tarau, P. (2016b). Infusing NLU into automatic question generation. In: the 9th International Natural Language Generation conference, pp. 51–60.

Miller, G A, Beckwith, R, Fellbaum, C, Gross, D, Miller, K J. (1990). Introduction to WordNet: An on-line lexical database. International Journal of Lexicography , 3 (4), 235–244.

Mitkov, R, & Ha, L A. (2003). Computer-aided generation of multiple-choice tests. In The HLT-NAACL 03 workshop on building educational applications using natural language processing, association for computational linguistics, pp. 17–22 .

Mitkov, R, Le An, H, Karamanis, N. (2006). A computer-aided environment for generating multiple-choice test items. Natural language engineering , 12 (2), 177–194. https://doi.org/10.1017/S1351324906004177 .

Montenegro, C S, Engle, V G, Acuba, M G J, Ferrenal, A M A. (2012). Automated question generator for Tagalog informational texts using case markers. In TENCON 2012-2012 IEEE region 10 conference, IEEE, pp. 1–5 . https://doi.org/10.1109/TENCON.2012.6412273 .

Mostow, J, & Chen, W. (2009). Generating instruction automatically for the reading strategy of self-questioning. In: the 14th international conference artificial intelligence in education, pp. 465–472.

Mostow, J, Beck, J, Bey, J, Cuneo, A, Sison, J, Tobin, B, Valeri, J. (2004). Using automated questions to assess reading comprehension, vocabulary, and effects of tutorial interventions. Technology Instruction Cognition and Learning , 2 , 97–134.

Mostow, J, Yt, Huang, Jang, H, Weinstein, A, Valeri, J, Gates, D. (2017). Developing, evaluating, and refining an automatic generator of diagnostic multiple choice cloze questions to assess children’s comprehension while reading. Natural Language Engineering , 23 (2), 245–294. https://doi.org/10.1017/S1351324916000024 .

Niraula, NB, & Rus, V. (2015). Judging the quality of automatically generated gap-fill question using active learning. In: the 10th workshop on innovative use of NLP for building educational applications, pp. 196–206.

Odilinye, L, Popowich, F, Zhang, E, Nesbit, J, Winne, PH. (2015). Aligning automatically generated questions to instructor goals and learner behaviour. In: the IEEE 9th international conference on semantic computing (ICS), pp. 216–223. https://doi.org/10.1109/ICOSC.2015.7050809 .

Olney, A M, Pavlik, P I, Maass, J K. (2017). Improving reading comprehension with automatically generated cloze item practice. In André, E., Baker, R., Hu, X., Rodrigo, M.M.T., du Boulay, B. (Eds.) Artificial intelligence in education . https://doi.org/10.1007/978-3-319-61425-0_22 (pp. 262–273). Cham: Springer International Publishing.

Papasalouros, A, & Chatzigiannakou, M. (2018). Semantic web and question generation: An overview of the state of the art. In: The international conference e-learning, pp. 189–192.

Papineni, K, Roukos, S, Ward, T, Zhu, WJ. (2002). BLEU: a method for automatic evaluation of machine translation. In: the 40th annual meeting on association for computational linguistics, Association for computational linguistics, pp. 311–318.

Park, J, Cho, H, Sg, Lee. (2018). Automatic generation of multiple-choice fill-in-the-blank question using document embedding. In Penstein Rosé, C., Martínez-Maldonado, R., Hoppe, H.U., Luckin, R., Mavrikis, M., Porayska-Pomsta, K., McLaren, B., du Boulay, B. (Eds.) Artificial intelligence in education (pp. 261–265). Cham: Springer International Publishing.

Patra, R, & Saha, SK. (2018a). Automatic generation of named entity distractors of multiple choice questions using web information Pattnaik, P.K., Rautaray, SS, Das, H, Nayak, J (Eds.), Springer, Berlin.

Patra, R, & Saha, SK. (2018b). A hybrid approach for automatic generation of named entity distractors for multiple choice questions. Education and Information Technologies pp. 1–21.

Polozov, O, O’Rourke, E, Smith, A M, Zettlemoyer, L, Gulwani, S, Popovic, Z. (2015). Personalized mathematical word problem generation. In The 24th international joint conference on artificial intelligence (IJCAI 2015), pp. 381–388 .

Qayyum, A, & Zawacki-Richter, O. (2018). Distance education in Australia, Europe and the americas, Springer, Berlin.

Rajpurkar, P, Zhang, J, Lopyrev, K, Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. In: the 2016 conference on empirical methods in natural language processing, pp. 2383–2392.

Rakangor, S, & Ghodasara, Y R. (2015). Literature review of automatic question generation systems. International Journal of Scientific and Research Publications , 5 (1), 2250–3153.

Reisch, J S, Tyson, J E, Mize, S G. (1989). Aid to the evaluation of therapeutic studies. Pediatrics , 84 (5), 815–827.

Rocha, OR, & Zucker, CF. (2018). Automatic generation of quizzes from DBpedia according to educational standards. In: the 3rd educational knowledge management workshop (EKM).

Rus, V, Wyse, B, Piwek, P, Lintean, M, Stoyanchev, S, Moldovan, C. (2012). A detailed account of the first question generation shared task evaluation challenge. Dialogue & Discourse , 3 (2), 177–204.

Rush, B R, Rankin, D C, White, B J. (2016). The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education , 16 (1), 250. https://doi.org/10.1186/s12909-016-0773-3 .

Santhanavijayan, A, Balasundaram, S, Narayanan, S H, Kumar, S V, Prasad, V V. (2017). Automatic generation of multiple choice questions for e-assessment. International Journal of Signal and Imaging Systems Engineering , 10 (1-2), 54–62.

Sarin, Y, Khurana, M, Natu, M, Thomas, A G, Singh, T. (1998). Item analysis of published MCQs. Indian Pediatrics , 35 , 1103–1104.

Satria, AY, & Tokunaga, T. (2017a). Automatic generation of english reference question by utilising nonrestrictive relative clause. In: the 9th international conference on computer supported education, pp. 379–386. https://doi.org/10.5220/0006320203790386 .

Satria, AY, & Tokunaga, T. (2017b). Evaluation of automatically generated pronoun reference questions. In: the 12th workshop on innovative use of NLP for building educational applications, pp. 76–85.

Serban, IV, García-Durán, A., Gulcehre, C, Ahn, S, Chandar, S, Courville, A, Bengio, Y. (2016). Generating factoid questions with recurrent neural networks: The 30M factoid question-answer corpus. ACL.

Seyler, D, Yahya, M, Berberich, K. (2017). Knowledge questions from knowledge graphs. In: The ACM SIGIR international conference on theory of information retrieval, pp. 11–18.

Shah, R, Shah, D, Kurup, L. (2017). Automatic question generation for intelligent tutoring systems. In: the 2nd international conference on communication systems, computing and it applications (CSCITA), pp. 127–132. https://doi.org/10.1109/CSCITA.2017.8066538 .

Shenoy, V, Aparanji, U, Sripradha, K, Kumar, V. (2016). Generating DFA construction problems automatically. In: The international conference on learning and teaching in computing and engineering (LATICE), pp. 32–37. https://doi.org/10.1109/LaTiCE.2016.8 .

Shirude, A, Totala, S, Nikhar, S, Attar, V, Ramanand, J. (2015). Automated question generation tool for structured data. In: International conference on advances in computing, communications and informatics (ICACCI), pp. 1546–1551. https://doi.org/10.1109/ICACCI.2015.7275833 .

Singhal, R, & Henz, M. (2014). Automated generation of region based geometric questions.

Singhal, R, Henz, M, Goyal, S. (2015a). A framework for automated generation of questions across formal domains. In: the 17th international conference on artificial intelligence in education, pp. 776–780.

Singhal, R, Henz, M, Goyal, S. (2015b). A framework for automated generation of questions based on first-order logic Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (Eds.), Springer International Publishing, Cham.

Singhal, R, Goyal, R, Henz, M. (2016). User-defined difficulty levels for automated question generation. In: the IEEE 28th international conference on tools with artificial intelligence (ICTAI), pp. 828–835. https://doi.org/10.1109/ICTAI.2016.0129 .

Song, L, & Zhao, L. (2016a). Domain-specific question generation from a knowledge base. Tech. rep.

Song, L, & Zhao, L. (2016b). Question generation from a knowledge base with web exploration. Tech. rep.

Soonklang, T, & Muangon, W. (2017). Automatic question generation system for English exercise for secondary students. In: the 25th international conference on computers in education.

Stasaski, K, & Hearst, MA. (2017). Multiple choice question generation utilizing an ontology. In: the 12th workshop on innovative use of NLP for building educational applications, pp. 303–312.

Susanti, Y, Iida, R, Tokunaga, T. (2015). Automatic generation of English vocabulary tests. In: the 7th international conference on computer supported education, pp. 77–87.

Susanti, Y, Nishikawa, H, Tokunaga, T, Hiroyuki, O. (2016). Item difficulty analysis of English vocabulary questions. In The 8th international conference on computer supported education (CSEDU 2016), pp. 267–274 .

Susanti, Y, Tokunaga, T, Nishikawa, H, Obari, H. (2017a). Controlling item difficulty for automatic vocabulary question generation. Research and Practice in Technology Enhanced Learning , 12 (1), 25. https://doi.org/10.1186/s41039-017-0065-5 .

Susanti, Y, Tokunaga, T, Nishikawa, H, Obari, H. (2017b). Evaluation of automatically generated English vocabulary questions. Research and Practice in Technology Enhanced Learning 12(1). https://doi.org/10.1186/s41039-017-0051-y .

Tamura, Y, Takase, Y, Hayashi, Y, Nakano, Y I. (2015). Generating quizzes for history learning based on Wikipedia articles. In Zaphiris, P., & Ioannou, A. (Eds.) Learning and collaboration technologies (pp. 337–346). Cham: Springer International Publishing.

Tarrant, M, Knierim, A, Hayes, S K, Ware, J. (2006). The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Education in Practice , 6 (6), 354–363. https://doi.org/10.1016/j.nepr.2006.07.002 .

Tarrant, M, Ware, J, Mohammed, A M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis. BMC Medical Education , 9 (1), 40. https://doi.org/10.1186/1472-6920-9-40 .

Thalheimer, W. (2003). The learning benefits of questions. Tech. rep., Work Learning Research. http://www.learningadvantage.co.za/pdfs/questionmark/LearningBenefitsOfQuestions.pdf .

Thomas, A, Stopera, T, Frank-Bolton, P, Simha, R. (2019). Stochastic tree-based generation of program-tracing practice questions. In: the 50th ACM technical symposium on computer science education, ACM, pp. 91–97.

Vie, J J, Popineau, F, Bruillard, É., Bourda, Y. (2017). A review of recent advances in adaptive assessment, Springer, Berlin.

Viera, A J, Garrett, J M, et al. (2005). Understanding interobserver agreement: the kappa statistic. Family Medicine , 37 (5), 360–363.

Vinu, EV, & Kumar, PS. (2015a). Improving large-scale assessment tests by ontology based approach. In: the 28th international florida artificial intelligence research society conference, pp. 457– 462.

Vinu, EV, & Kumar, PS. (2015b). A novel approach to generate MCQs from domain ontology: Considering DL semantics and open-world assumption. Web Semantics: Science, Services and Agents on the World Wide Web , 34 , 40–54. https://doi.org/10.1016/j.websem.2015.05.005 .

Vinu, EV, & Kumar, PS. (2017a). Automated generation of assessment tests from domain ontologies. Semantic Web Journal , 8 (6), 1023–1047. https://doi.org/10.3233/SW-170252 .

Vinu, EV, & Kumar, PS. (2017b). Difficulty-level modeling of ontology-based factual questions. Semantic Web Journal In press.

Vinu, E V, Alsubait, T, Kumar, PS. (2016). Modeling of item-difficulty for ontology-based MCQs. Tech. rep.

Wang, K, & Su, Z. (2016). Dimensionally guided synthesis of mathematical word problems. In: the 25th International Joint Conference on Artificial Intelligence (IJCAI), pp. 2661–2668.

Wang, K, Li, T, Han, J, Lei, Y. (2012). Algorithms for automatic generation of logical questions on mobile devices. IERI Procedia , 2 , 258–263. https://doi.org/10.1016/j.ieri.2012.06.085 .

Wang, Z, Lan, AS, Nie, W, Waters, AE, Grimaldi, PJ, Baraniuk, RG. (2018). QG-net: a data-driven question generation model for educational content. In: the 5th Annual ACM Conference on Learning at Scale, pp. 15–25.

Ware, J, & Vik, T. (2009). Quality assurance of item writing: During the introduction of multiple choice questions in medicine for high stakes examinations. Medical Teacher , 31 (3), 238–243. https://doi.org/10.1080/01421590802155597 .

Webb, N L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education . Tech. rep.: National Institute for Science Education.

Welbl, J, Liu, NF, Gardner, M. (2017). Crowdsourcing multiple choice science questions. In: the 3rd workshop on noisy user-generated text, pp. 94–106.

Wita, R, Oly, S, Choomok, S, Treeratsakulchai, T, Wita, S. (2018). A semantic graph-based Japanese vocabulary learning game. In Hancke, G., Spaniol, M., Osathanunkul, K., Unankard, S., Klamma, R. (Eds.) Advances in web-based learning – ICWL , (Vol. 2018 pp. 140–145). Cham: Springer International Publishing.. https://doi.org/10.1007/978-3-319-96565-9_14

Yaneva, V, & et al. (2018). Automatic distractor suggestion for multiple-choice tests using concept embeddings and information retrieval. In: the 13th workshop on innovative use of NLP for building educational applications, pp. 389–398.

Yao, X, Bouma, G, Zhang, Y. (2012). Semantics-based question generation and implementation. Dialogue & Discourse , 3 (2), 11–42.

Zavala, L, & Mendoza, B. (2018). On the use of semantic-based AIG to automatically generate programming exercises. In: the 49th ACM technical symposium on computer science education, ACM, pp. 14–19.

Zhang, J, & Takuma, J. (2015). A Kanji learning system based on automatic question sentence generation. In: 2015 international conference on asian language processing (IALP), pp. 144–147. https://doi.org/10.1109/IALP.2015.7451552 .

Zhang, L. (2015). Biology question generation from a semantic network . PhD thesis: Arizona State University.

Zhang, L, & VanLehn, K. (2016). How do machine-generated questions compare to human-generated questions?. Research and Practice in Technology Enhanced Learning, 11(7). https://doi.org/10.1186/s41039-016-0031-7 .

Zhang, T, Quan, P, et al. (2018). Domain specific automatic Chinese multiple-type question generation. In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE, pp. 1967–1971 . https://doi.org/10.1109/BIBM.2018.8621162 .

Download references

Author information

Authors and affiliations.

Department of Computer Science, The University of Manchester, Manchester, UK

Ghader Kurdi, Jared Leo, Bijan Parsia & Uli Sattler

Umm Al-Qura University, Mecca, Saudi Arabia

Salam Al-Emari

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ghader Kurdi .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Excluded Studies

Publication venues, active research groups, summary of included studies, quality assessment, rights and permissions.

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Kurdi, G., Leo, J., Parsia, B. et al. A Systematic Review of Automatic Question Generation for Educational Purposes. Int J Artif Intell Educ 30 , 121–204 (2020). https://doi.org/10.1007/s40593-019-00186-y

Download citation

Published : 21 November 2019

Issue Date : March 2020

DOI : https://doi.org/10.1007/s40593-019-00186-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Automatic question generation
• Semantic Web
• Natural language processing
• Natural language generation
• Difficulty prediction
• Find a journal
• Publish with us
• Track your research