margin of error in research paper

• Foundations
• Write Paper

Search form

• Experiments
• Anthropology
• Self-Esteem
• Social Anxiety
• Statistics >
• Margin of Error

Margin of Error (Statistics)

In statistics margin of error plays a very important role in many social science experiments, surveys, etc.

This article is a part of the guide:

• Significance 2
• Sample Size
• Experimental Probability
• Cronbach’s Alpha
• Systematic Error

Browse Full Outline

• 1 Inferential Statistics
• 2.1 Bayesian Probability
• 3.1.1 Significance 2
• 3.2 Significant Results
• 3.3 Sample Size
• 3.4 Margin of Error
• 3.5.1 Random Error
• 3.5.2 Systematic Error
• 3.5.3 Data Dredging
• 3.5.4 Ad Hoc Analysis
• 3.5.5 Regression Toward the Mean
• 4.1 P-Value
• 4.2 Effect Size
• 5.1 Philosophy of Statistics
• 6.1.1 Reliability 2
• 6.2 Cronbach’s Alpha

The margin of error determines how reliable the survey is or how reliable the results of the experiment are.

Any survey takes a sample population from the whole population and then generalizes the results to the whole population. This invariably leads to a possibility of error because the whole can never be accurately described by a part of it.

This is captured in statistics as margin of error. The higher the margin of error, the less likely it is that the results of the survey are true for the whole population.

In statistics margin of error is related to the confidence interval as being equal to half the interval length. This means higher the confidence interval, higher the margin of error for the same set of data. This is expected because to get a higher confidence interval, one usually needs higher data points. It is also quite expected that as the number of samples increases, the margin of error decreases.

The margin of error is usually expressed as a percentage, but in some cases, may also be expressed as the absolute number. In statistics margin of error makes the most sense for normally distributed data, but can still be a useful parameter otherwise.

With margin of error, the statistics represented by the survey make sense. If a survey finds that 36% of the respondents watch television while eating lunch, the information is incomplete. When the margin of error is specified, say, 4%, then this means the 36% should be interpreted as 32-40%. This makes complete sense.

However, there are scenarios in statistics when margin of error is unable to take care of the error of the survey. This happens when the survey has poorly designed questions or the respondents have other bias in their answering or are lying for some reason. Also, if the sample is not chosen to be representative of the whole population, errors beyond the margin of error may occur.

• Psychology 101
• Flags and Countries
• Capitals and Countries

Siddharth Kalla (Jan 9, 2009). Margin of Error (Statistics). Retrieved Jun 01, 2024 from Explorable.com: https://explorable.com/statistics-margin-of-error

You Are Allowed To Copy The Text

The text in this article is licensed under the Creative Commons-License Attribution 4.0 International (CC BY 4.0) .

This means you're free to copy, share and adapt any parts (or all) of the text in the article, as long as you give appropriate credit and provide a link/reference to this page.

That is it. You don't need our permission to copy the article; just include a link/reference back to this page. You can use it freely (with some kind of link), and we're also okay with people reprinting in publications like books, blogs, newsletters, course-material, papers, wikipedia and presentations (with clear attribution).

Want to stay up to date? Follow us!

Save this course for later.

Don't have time for it all now? No problem, save it as a course and come back to it later.

Footer bottom

• Privacy Policy

• Subscribe to our RSS Feed
• Like us on Facebook
• Follow us on Twitter

AP® Statistics

Margin of error: what to know for ap® statistics.

• The Albert Team
• Last Updated On: March 1, 2022

Introduction

While you are learning statistics, you will often have to focus on a sample rather than the entire population. This is because it is extremely costly, difficult and time-consuming to study the entire population. The best you can do is to take a random sample from the population – a sample that is a ‘true’ representative of it. You then carry out some analysis using the sample and make inferences about the population.

Since the inferences are made about the population by studying the sample taken, the results cannot be entirely accurate. The degree of accuracy depends on the sample taken – how the sample was selected, what the sample size is, and other concerns. Common sense would say that if you increase the sample size, the chances of error will be less because you are taking a greater proportion of the population. A larger sample is likely to be a closer representative of the population than a smaller one.

Let’s consider an example. Suppose you want to study the scores obtained in an examination by students in your college. It may be time-consuming for you to study the entire population, i.e. all students in your college. Hence, you take out a sample of, say, 100 students and find out the average scores of those 100 students. This is the sample mean. Now, when you use this sample mean to infer about the population mean, you won’t be able to get the exact population means. There will be some “margin of error”.

You will now learn the answers to some important questions: What is margin of error, what are the method of calculating margins of error, how do you find the critical value, and how to decide on t-score vs z-scores. Thereafter, you’ll be given some margin of error practice problems to make the concepts clearer.

What is Margin of Error?

The margin of error can best be described as the range of values on both sides (above and below) the sample statistic. For example, if the sample average scores of students are 80 and you make a statement that the average scores of students are 80 ± 5, then here 5 is the margin of error.

Calculating Margins of Error

For calculating margins of error, you need to know the critical value and sample standard error. This is because it’s calculated using those two pieces of information.

The formula goes like this:

margin of error = critical value * sample standard error.

How do you find the critical value, and how to calculate the sample standard error? Below, we’ll discuss how to get these two important values.

How do You find the Critical Value?

For finding critical value, you need to know the distribution and the confidence level. For example, suppose you are looking at the sampling distribution of the means. Here are some guidelines.

• If the population standard deviation is known, use z distribution.
• If the population standard deviation is not known, use t distribution where degrees of freedom = n-1 ( n is the sample size). Note that for other sampling distributions, degrees of freedom can be different and should be calculated differently using appropriate formula.
• If the sample size is large, then use z distribution (following the logic of Central Limit Theorem).

It is important to know the distribution to decide what to use – t-scores vs z-scores.

Caution – when your sample size is large and it is not given that the distribution is normal, then by Central Limit Theorem, you can say that the distribution is normal and use z-score. However, when the sample size is small and it is not given that the distribution is normal, then you cannot conclude anything about the normality of the distribution and neither z-score nor t-score can be used.

When finding the critical value, confidence level will be given to you. If you are creating a 90% confidence interval, then confidence level is 90%, for 95% confidence interval, the confidence level is 95%, and so on.

Here are the steps for finding critical value:

Step 1: First, find alpha (the level of significance). \alpha =1 – Confidence level.

Step 2: Find the critical probability p* . Critical probability will depend on whether we are creating a one-sided confidence interval or a two-sided confidence interval.

Then you need to decide on using t-scores vs z-scores. Find a z-score having a cumulative probability of p* . For a t-statistic, find a t-score having a cumulative probability of p*  and the calculated degrees of freedom. This will be the critical value. To find these critical values, you should use a calculator or respective statistical tables.

Sample Standard Error

Sample standard error can be calculated using population standard deviation or sample standard deviation (if population standard deviation is not known). For sampling distribution of means:

Let sample standard deviation be denoted by s , population standard deviation is denoted by \sigma and sample size be denoted by n .

\text {Sample standard error}=\dfrac { \sigma }{ \sqrt { n } } , if \sigma is known

\text {Sample standard error}=\dfrac { s }{ \sqrt { n } } , if \sigma is not known

Depending on the sampling distributions, the sample standard error can be different.

Having looked at everything that is required to create the margin of error, you can now directly calculate a margin of error using the formula we showed you earlier:

Margin of error = critical value * sample standard error.

Some Relationships

1. confidence level and marginal of error.

As the confidence level increases, the critical value increases and hence the margin of error increases. This is intuitive; the price paid for higher confidence level is that the margin of errors increases. If this was not so, and if higher confidence level meant lower margin of errors, nobody would choose a lower confidence level. There are always trade-offs!

2. Sample standard deviation and margin of error

Sample standard deviation talks about the variability in the sample. The more variability in the sample, the higher the chances of error, the greater the sample standard error and margin of error.

3. Sample size and margin of error

This was discussed in the Introduction section. It is intuitive that a greater sample size will be a closer representative of the population than a smaller sample size. Hence, the larger the sample size, the smaller the sample standard error and therefore the smaller the margin of error.

Margin of Error Practice Problems

25 students in their final year were selected at random from a high school for a survey. Among the survey participants, it was found that the average GPA (Grade Point Average) was 2.9 and the standard deviation of GPA was 0.5. What is the margin of error, assuming 95% confidence level? Give correct interpretation.

Step 1 : Identify the sample statistic.

Since you need to find the confidence interval for the population mean, the sample statistic is the sample mean which is the average GPA = 2.9.

Step 2 : Identify the distribution – t, z, etc. – and find the critical value based on whether you need a one-sided confidence interval or a two-sided confidence interval.

Since population standard deviation is not known and the sample size is small, use a t distribution.

\text {Degrees of freedom}=n-1=25-1=24 .

Let the critical probability be p* .

For two-sided confidence interval,

p*=1-\dfrac { \alpha }{ 2 } =1-\dfrac { 0.05 }{ 2 } =0.975 .

The critical t value for cumulative probability of 0.975 and 24 degrees of freedom is 2.064.

Step 3: Find the sample standard error.

Step 4 : Find margin of error using the formula:

Margin of error = critical value * sample standard error

= 2.064 * 0.1 = 0.2064

Interpretation: For a 95% confidence level, the average GPA is going to be 0.2064 points above and below the sample average GPA of 2.9.

400 students in Princeton University are randomly selected for a survey which is aimed at finding out the average time students spend in the library in a day. Among the survey participants, it was found that the average time spent in the university library was 45 minutes and the standard deviation was 10 minutes. Assuming 99% confidence level, find the margin of error and give the correct interpretation of it.

Since you need to find the confidence interval for the population mean, the sample statistic is the sample mean which is the mean time spent in the university library = 45 minutes.

Step 2 : Identify the distribution – t, z, etc. and find the critical value based on whether the need is a one-sided confidence interval or a two-sided confidence interval.

The population standard deviation is not known, but the sample size is large. Therefore, use a z (standard normal) distribution.

p*=1-\dfrac { \alpha }{ 2 } =1-\dfrac { 0.01 }{ 2 } =0.995 .

The critical z value for cumulative probability of 0.995 (as found from the z tables) is 2.576.

Step 3 : Find the sample standard error.

= 2.576 * 0.5 = 1.288

Interpretation: For a 99% confidence level, the mean time spent in the library is going to be 1.288 minutes above and below the sample mean time spent in the library of 45 minutes.

Consider a similar set up in Example 1 with slight changes. You randomly select X students in their final year from a high school for a survey. Among the survey participants, it was found that the average GPA (Grade Point Average) was 3.1 and the standard deviation of GPA was 0.7. What should be the value of X (in other words, how many students you should select for the survey) if you want the margin of error to be at most 0.1? Assume 95% confidence level and normal distribution.

Step 1 : Find the critical value.

The critical z value for cumulative probability of 0.975 is 1.96.

Step 3: Find the sample standard error in terms of X .

Step 4 : Find X using margin of error formula:

This gives X=188.24 .

Thus, a sample of 189 students should be taken so that the margin of error is at most 0.1.

The margin of error is an extremely important concept in statistics. This is because it is difficult to study the entire population and the sampling is not free from sampling errors. The margin of error is used to create confidence intervals, and most of the time the results are reported in the form of a confidence interval for a population parameter rather than just a single value. In this article, you made a beginning by learning answering questions like what is margin of error, what is the method of calculating margins of errors, and how to interpret these calculations. You also learned to decide whether to use t-scores vs z-scores and gained information about finding critical values. Now you know how to use margin of error for constructing confidence intervals, which are widely used in statistics and econometrics.

Let’s put everything into practice. Try this Statistics practice question:

Looking for more statistics practice.

You can find thousands of practice questions on Albert.io. Albert.io lets you customize your learning experience to target practice where you need the most help. We’ll give you challenging practice questions to help you achieve mastery in Statistics.

Start practicing here .

Are you a teacher or administrator interested in boosting Statistics student outcomes?

Learn more about our school licenses here .

Interested in a school license?​

Popular posts.

AP® Score Calculators

Simulate how different MCQ and FRQ scores translate into AP® scores

AP® Review Guides

The ultimate review guides for AP® subjects to help you plan and structure your prep.

Core Subject Review Guides

Review the most important topics in Physics and Algebra 1 .

SAT® Score Calculator

See how scores on each section impacts your overall SAT® score

ACT® Score Calculator

See how scores on each section impacts your overall ACT® score

Grammar Review Hub

Comprehensive review of grammar skills

AP® Posters

Download updated posters summarizing the main topics and structure for each AP® exam.

Root out friction in every digital experience, super-charge conversion rates, and optimize digital self-service

Uncover insights from any interaction, deliver AI-powered agent coaching, and reduce cost to serve

Increase revenue and loyalty with real-time insights and recommendations delivered to teams on the ground

Know how your people feel and empower managers to improve employee engagement, productivity, and retention

Take action in the moments that matter most along the employee journey and drive bottom line growth

Whatever they’re are saying, wherever they’re saying it, know exactly what’s going on with your people

Get faster, richer insights with qual and quant tools that make powerful market research available to everyone

Run concept tests, pricing studies, prototyping + more with fast, powerful studies designed by UX research experts

Track your brand performance 24/7 and act quickly to respond to opportunities and challenges in your market

Explore the platform powering Experience Management

• Free Account
• For Digital
• For Customer Care
• For Human Resources
• For Researchers
• Financial Services
• All Industries

Popular Use Cases

• Customer Experience
• Employee Experience
• Net Promoter Score
• Voice of Customer
• Customer Success Hub
• Product Documentation
• Training & Certification
• XM Institute
• Popular Resources
• Customer Stories
• Artificial Intelligence
• Market Research
• Partnerships
• Marketplace

The annual gathering of the experience leaders at the world’s iconic brands building breakthrough business results, live in Salt Lake City.

• English/AU & NZ
• Español/Europa
• Español/América Latina
• Português Brasileiro
• REQUEST DEMO
• Experience Management
• Survey Data Analysis & Reporting
• Margin of error
• Margin of error calculator

What is margin of error?

When is margin of error used, how to calculate margin of error, how qualtrics can help, try qualtrics for free, margin of error guide & calculator.

11 min read Understand the accuracy of your survey data in seconds with our handy Margin of Error Calculator. Learn what margin of error is, when it should be used and what affects it.

Written by : Will Webster

Reviewed by : Aaron Carpenter 

When essential business decisions are being made using survey data, you need to know if that data is accurate and reliable. But can any market research – or any research in general – truly reflect a whole population?

The answer is typically no – it’s almost impossible to find a sample that 100% matches the characteristics of an entire population. Which is why it’s so important to build margin of error into your research and data.

But what exactly is margin of error? How much can or should it impact the level of confidence you have in your survey results? And most importantly, how do you calculate it?

Free eBook: 2024 global market research trends report

Try our Margin of Error Calculator

Input your survey data into our easy-to-use calculator, or experiment with the numbers to see how they impact margin of error.

If you want to learn how margin of error is calculated, skip ahead to our breakdown of the margin of error formula

Margin of Error Calculator

Function calculatess() { var confidencelevel = document.getelementbyid("confidencelevel").value; var samplesize = document.getelementbyid("samplesize").value; var proportionpercentage = document.getelementbyid( "proportionpercentage" ).value; var populationsize = document.getelementbyid("populationsize").value; var per = proportionpercentage / 100; var notop = samplesize; var p = math.sqrt((1 - per) * per); var nobottom = populationsize; var no = nobottom - notop; var marginoferror = ( confidencelevel * (p / math.sqrt(((nobottom - 1) * notop) / no)) * 100 ).tofixed(1); document.getelementbyid("marginoferror").value = marginoferror + "%"; } calculatess(); document .getelementbyid("samplesize") .addeventlistener("change", calculatess); document .getelementbyid("proportionpercentage") .addeventlistener("change", calculatess); document .getelementbyid("populationsize") .addeventlistener("change", calculatess); document.addeventlistener("keyup", calculatess);.

Margin of error is used in research to determine the precision of a result, such as a poll , survey , or a scientific study.

Expressed as +/- percentage points, margin of error tells you to what degree your research results may differ from the real-world results, revealing how different – more and less – the stated percentage may be from reality. A smaller margin of error is better as it suggests the survey’s results are more precise.

Putting this into practice, if researchers estimate that 52% of people prefer a certain product with a +/-3 percentage point margin of error at a desired confidence level of 95%, the actual percentage of people in the overall population who prefer the product is between 49% and 55% 95 times out of 100.

In this specific example, the margin of error calculation is especially valuable because it shows us that the people who prefer this product may not actually be in the majority.

Margin of error is calculated from a number of variables: your desired precision (which is known as the confidence level and is typically 95%), your sample proportion, and your sample size. These statistics are all explained in further detail when we look at how to calculate margin of error

What is a confidence interval?

A confidence interval is a range that we think contains the true value of something we’re measuring in a population. We apply margin of error to it to help us understand how precise our estimate is.

A confidence interval comprises a lower and upper limit that the true value of a population parameter is deemed to fall between, with a specified level of confidence that’s usually 95%. For example, a 95% confidence interval for the proportion of people who prefer a certain product might be between 49% and 55%, which means you’re 95% confident that the true mean value for the total population lies within that range.

Margin of error is used when you have a random or probability sample. That means the survey respondents have been selected at random from your total population.

However, it won’t be relevant to your market research data if your sample has been selected in a non-random way, like if you’ve used an opt-in research panel . Your participants may have been selected because they have particular characteristics, or have volunteered for the panel in return for benefits, and therefore aren’t randomly selected from the population size at large.

Margin of error for a proportion is calculated using the following formula:

• z is the z-score for your selected confidence level
• p is the sample proportion
• n is the sample size

Confidence level indicates how certain you are that your sample reflects the total population. The most commonly used confidence level is 95%. A 95% confidence level means that if you took a random sample of this population 100 times, you would expect the value you’ve got to be within the confidence interval 95% of the time. The z-score for a 95% confidence level is 1.96. Confidence levels of 90% and 99% are also used, with respective z-scores of 1.65 and 2.58.

The sample proportion is the percentage of the sample that has the characteristic you’re interested in, such as the percentage of respondents who were very satisfied with your product. It’s a decimal number representing a percentage, so while you’re doing the calculation it’s expressed in hundredths: if our proportion is 5% of respondents, it would be 0.05.

Finally, the sample size is the total number of people you’re surveying, or your number of completed responses.

Find the correct sample size with our Sample Size Calculator

How to calculate margin of error: Example question

Imagine you’re a business surveying your current customers . You’ve run a survey with a randomly selected sample of 1,000 people from your CRM that represents your target market.

The results tell you that of these 1,000 customers, 52% (520 people) are happy with their latest purchase, but 48% (480 people) are not – yikes. What is your survey’s margin of error?

The 520 customers who are happy with their latest purchase represent our sample proportion. To use this in the margin of error formula, we need to express it as a decimal representing a percentage of our total sample size. To do that we just need to divide 520 by 1,000, which gives us 0.52 – our p value.

We’ll assume a 95% level of confidence, so the z-score is 1.96.

How to calculate margin of error: Step-by-step guide

Here’s how to use the margin of error for a proportion formula, using the numbers from our previous section.

• Subtract p (0.52) from 1, which gives 0.48
• Multiply 0.48 by p, which equals 0.2496
• Divide 0.2496 by n (1,000), giving 0.0002496
• Calculate the square root of 0.0002496, which is 0.0157987
• Multiply 0.0157987 by the z-score (1.96), which equals 0.0309654
• Express that decimal as a percentage, which is 3.1% when rounded up

We can now report with 95% confidence that 52% of customers were happy with their latest purchase, with a margin of error of +/- 3.1%.

Considerations when using the margin of error for a proportion formula

There are a couple of conditions to note when using this formula to calculate margin of error:

• n x p must equal 10 or more
• n x (1-p) must equal 10 or more

But these won’t impact the majority of survey results. That’s because survey research usually involves high numbers of people in a sample. So, unless your sample is a smaller group, or the proportion within your sample is very small, there won’t be a problem. If you’re getting numbers below 10 for either of these checks, you may need to increase your sample size.

What affects margin of error?

Margin of error can be influenced by a range of different values.

In the below table, we can see how margin of error shifts when sample size and sample proportion change. Here we’re assuming that a simple random sample is being surveyed, that the full population size is over 20,000, that power is 80% and a 95% confidence level.

How sample size affects margin of error

The general rule is that the larger the sample size, the smaller the margin of error, and vice versa.

When we take a larger sample size, we increase the representation of the population in the sample. This means that the sample proportion estimate is more likely to be closer to the true population proportion, which reduces the margin error.

A smaller sample size, on the other hand, is less representative of the population, which increases the margin error.

How sample proportion affects margin of error

With sample proportion, the margin of error increases as the proportion gets closer to 50%, and decreases when the proportion is closer to 0% or 100% – as we can see in the above table. This happens because the standard error is largest when the sample proportion is 50%.

However, the effect of sample size on margin of error is stronger than the effect of sample proportion, meaning that increasing the sample size will still decrease the margin of error, regardless of the sample proportion.

How population variability affects margin of error

Population variability, or population standard deviation, is a measure of how spread out the data values are in a population. A larger standard deviation means that the data values are more spread out, and a smaller standard deviation means that the data values are more tightly clustered around the mean.

The larger the population variability, the larger the margin of error; the smaller the population variability, the smaller the margin of error. If the population is more variable, you need a larger sample size to achieve that smaller margin of error.

No matter the survey, research project or poll, Qualtrics CoreXM has the tools and capabilities you need to derive insight from data — and go from insight to action.

Used by more than 13,000 customers, Qualtrics CoreXM brings your survey, research, and polls into one platform, rather than multiple tools, enabling you to scale your activities, improve data quality, and uncover new opportunities.

With CoreXM, get insights faster through automation, reduce project costs and admin by consolidating your tools, and increase project impact by leveraging best-in-class survey methodologies and tools.

From simple polling and internal feedback surveys for strategy to gleaning customer , product , brand , and market insights, Qualtrics CoreXM empowers every department to carry out and benefit from research.

And if you want to get even more from your research, take advantage of Qualtrics’ Research Services and a network of experienced partners. Through trainer experts, unlock cost and time savings, while benefiting from flexible service options that let you choose how much or how little support you need.

Related resources

Analysis & Reporting

Data Saturation In Qualitative Research 8 min read

Thematic analysis 11 min read, behavioral analytics 12 min read, statistical significance calculator: tool & complete guide 18 min read, regression analysis 19 min read, data analysis 31 min read, social media analytics 13 min read, request demo.

Ready to learn more about Qualtrics?

Teach yourself statistics

Margin of Error

In a confidence interval , the range of values above and below the sample statistic is called the margin of error .

For example, suppose we wanted to know the percentage of adults that exercise daily. We could devise a sample design to ensure that our sample estimate will not differ from the true population value by more than, say, 5 percent 90 percent of the time (the confidence level ). In this example, the margin of error would be 5 percent.

How to Compute the Margin of Error

The margin of error can be defined by either of the following equations.

Margin of error = Critical value x Standard deviation of the statistic Margin of error = Critical value x Standard error of the statistic

If you know the standard deviation of the statistic, use the first equation to compute the margin of error. Otherwise, use the second equation. In a previous lesson, we described how to compute the standard deviation and standard error .

How to Find the Critical Value

The critical value is a factor used to compute the margin of error. This section describes how to find the critical value, when the sampling distribution of the statistic is normal or nearly normal.

When the sampling distribution is nearly normal, the critical value can be expressed as a t score or as a z-score . To find the critical value, follow these steps.

• Compute alpha (α): α = 1 - (confidence level / 100)
• Find the critical probability (p*): p* = 1 - α/2
• To express the critical value as a z-score, find the z-score having a cumulative probability equal to the critical probability (p*).
• Find the degrees of freedom (DF). When estimating a mean score or a proportion from a single sample, DF is equal to the sample size minus one. For other applications, the degrees of freedom may be calculated differently. We will describe those computations as they come up.
• The critical t statistic (t*) is the t statistic having degrees of freedom equal to DF and a cumulative probability equal to the critical probability (p*).

T-Score vs. Z-Score

Should you express the critical value as a t statistic or as a z-score? One way to answer this question focuses on the population standard deviation.

• If the population standard deviation is known, use the z-score.
• If the population standard deviation is unknown, use the t statistic.

Another approach focuses on sample size.

• If the sample size is large, use the z-score. (The central limit theorem provides a useful basis for determining whether a sample is "large".)
• If the sample size is small, use the t statistic.

In practice, researchers employ a mix of the above guidelines. On this site, we use z-scores when the population standard deviation is known and the sample size is large. Otherwise, we use t statistics, unless the sample size is small and the underlying distribution is not normal.

Warning: If the sample size is small and the population distribution is not normal, we cannot be confident that the sampling distribution of the statistic will be normal. In this situation, neither the t statistic nor the z-score should be used to compute critical values.

You can use the Normal Distribution Calculator to find the critical z-score, and the t Distribution Calculator to find the critical t statistic. You can also use a graphing calculator or standard statistical tables (found in the appendix of most introductory statistics texts).

Test Your Understanding

Nine hundred (900) high school freshmen were randomly selected for a national survey. Among survey participants, the mean grade-point average (GPA) was 2.7, and the standard deviation was 0.4. What is the margin of error, assuming a 95% confidence level?

I. When the margin of error is small, the confidence level is high. II. When the margin of error is small, the confidence level is low. III. A confidence interval is a type of point estimate. IV. A population mean is an example of a point estimate.

(A) 0.013 (B) 0.025 (C) 0.500 (D) 1.960 (E) None of the above.

The correct answer is (B). To compute the margin of error, we need to find the critical value and the standard error of the mean. To find the critical value, we take the following steps.

α = 1 - (confidence level / 100)

α = 1 - 0.95 = 0.05

p* = 1 - α/2

p* = 1 - 0.05/2 = 0.975

df = n - 1 = 900 -1 = 899

• Find the critical value. Since we don't know the population standard deviation, we'll express the critical value as a t statistic. For this problem, it will be the t statistic having 899 degrees of freedom and a cumulative probability equal to 0.975. Using the t Distribution Calculator , we find that the critical value is about 1.96.

Next, we find the standard error of the mean, using the following equation:

SE x = s / sqrt( n )

SE x = 0.4 / sqrt( 900 ) = 0.4 / 30 = 0.013

And finally, we compute the margin of error (ME).

ME = Critical value x Standard error

ME = 1.96 * 0.013 = 0.025

This means we can be 95% confident that the mean grade point average in the population is 2.7 plus or minus 0.025, since the margin of error is 0.025.

Note: The larger the sample size, the more closely the t distribution looks like the normal distribution. For this problem, since the sample size is very large, we would have found the same result with a z-score as we found with a t statistic. That is, the critical value would still have been about 1.96. The choice of t statistic versus z-score does not make much practical difference when the sample size is very large.

Understanding margin of error in surveys and the factors that influence it

Margin of error is a term commonly used in market research but often not very well understood. 

However, it has a significant impact on the accuracy of the conclusions you can draw from a survey.

The concept behind it is nuanced and slightly complex so read on as we attempt to simplify it for you! 

While we won't dive into how to calculate the margin of error (there are quite a few online calculators available without having to work it out yourself), we hope that at the end of this piece you have a better understanding of what margin of error is and what are the factors that affect it.

So what is margin of error?

Simply put, margin of error tells you how precise are the results you obtain from a survey.  Or in other words, as the name suggests, how much of a room there is for error. 

But why should there be any room for error in the first place?

When you conduct a survey, you are only talking to a subset of the population you want to draw conclusions about. 

For instance, if you want to know what percentage of a country owns dogs, you might survey only a n = 1000 sample of respondents rather than the entire population. 

It is reasonable to assume that the results you get from this survey of n = 1000 respondents won’t exactly match the “true” results you would get if you surveyed the entire population. 

This is where margin of error comes in. 

The margin of error tells you how close you can expect your results to be compared to the “true” population value . For instance, if your results indicate that 20% of the population owns a dog in a survey with a margin of error of 3%, it means your results can be plus or minus 3 percentage points away from the “true” population value i.e. within the range of 17% to 23% . 

The narrower the margin of error the more certain you are that the results of your survey are reflective of the opinions and behaviour of the overall target population.

Now how do you work towards your desired margin of error? For that you need to know what factors affect it. 

Factors that influence margin of error

Margin of error is a derived metric that depends on three factors  : population size, sample size, and confidence level . We elaborate on each of these below.

1. Population size ‍

The term population refers to the entire target group of people you want to study with your survey. 

In the above example of dog ownership, the population of interest is the entire population of a country. 

However,  if in a survey you want to study the opinions or behaviour of a subgroup, say millennials in a country, the population in this example would be the total number of millennials in that market. Margin of error may vary depending on the population size your survey aims to study.

2. Sample size ‍

While the population size is a fixed number beyond your control, sample size is the most important factor that researchers have to make a decision about while designing a survey because it has a significant impact on the margin of error.

There is an inverse relationship between the margin of error and the sample size : as the sample size increases, the margin of error decreases . And this is logical because the more people you survey, the more confident you can be that your results are closer to the “true” population value (keeping in mind that the survey does not have other biases of course).

However, an important point to take note of is that while the reduction in margin of error is considerable as you increase the sample size till n = 1000, there is a diminishing return when you go further up. It is often not worthwhile for researchers to devote additional time and money to collect more than n= 1000 samples as the margin of error does not considerably decrease below 3% after that.

‍ For a more detailed lowdown on how to decide on an optimal sample size for your survey, you may refer to this article.

3. Confidence level ‍

Confidence level denotes the probability with which the range of values estimated by the margin of error consists of the “true” population value. 

This means that even though the statistical calculations behind margin of error will produce a range of values which includes the “true” value, there is no 100% guarantee that the calculation is accurate all the time. There is a slight chance that the range may not include the “true” value.

The confidence level tells you how confident you can be that your margin of error range will include the true result.

You may have noticed that when reporting the margin of error, there is always a confidence level attached to it. For example :  the margin of error for this survey is 5% at the 95% confidence level . This means that if you were to conduct the survey 100 times, 95 of those times you would expect the result to be within 5 percentage points of the true population value. 

While 95% is the typical choice of a confidence level in market research, it is not uncommon to see a 90% confidence level or sometimes lower. The question you might ask is : why not always go for a higher confidence level if it gives you more certainty? The reason being it comes at a cost. 

To give you an example, if you surveyed n = 100 respondents and find that 20% of them are smokers, the margin of error is 6% at an 80% confidence level. In other words, the “true” percentage of smokers can range between 14% and 26%. However, if you change the confidence level to 95%, the margin of error increases to 10%, making the error range wider. If you want to bring down the margin of error to 6% at a confidence level of 95%, the solution would be to recruit more respondents. By bumping up the sample size to n = 230, you can now achieve a 6% margin of error at 95% confidence level. While this may be feasible in some instances, additional recruitment of respondents may come at an increased cost and time. 

To summarise, a lower confidence level gives you a lower margin of error, but with a higher probability that the range doesn’t consist of the actual value. A higher confidence level leads to a wider margin of error at the same sample size. The higher the confidence level, the wider the margin of error because it allows you to be more confident that the “true” population value lies within the +/- percentage  interval. ‍

Data collected from a survey will never perfectly represent the behaviour and opinions of the target population as a whole. How much room there is for error is contingent on the three factors outlined above :  population size, sample size, and confidence level . 

Ideally, the margin of error should not be an afterthought that’s calculated post survey fieldwork. It is good to design the study keeping in mind the level of accuracy that is desirable and sample size is the most important determinant of the margin of error. 

If cost, time, and feasibility are not barriers, researchers often aim for higher sample sizes close to n = 1000 so that the margin of error is low. However, if there are resource and operational constraints, one has to often work with lower samples, thus increasing the margin of error.

It is important to keep in mind the objective of the study. If it is crucial to get as precise results as possible because of important implications (e.g., election polls, government-led surveys with policy implications) then it is imperative that you keep the margin of error as low as possible. However, sometimes surveys are used just to get a rough sense of responses or opinions (e.g., customer feedback surveys, feedback on early stage product designs) where any feedback is valuable and representativeness of the entire population is not of utmost importance - here one may be comfortable with a higher margin of error.

Milieu Insight, an award-winning online survey tool and market research company , champions businesses in leveraging data for success.

Exploring the evolution of online brand research, our brand health research guide emphasizes engaging diverse audiences and employing adaptable methodologies to yield comprehensive insights.

Optimal survey length: How long survey lengths can affect data quality

How grid questions can compromise your survey data.

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case NPS+ Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research

Margin of Error: Definition + Easy Calculation with Examples

The margin of error is an essential concept for understanding the accuracy and reliability of survey data. In this article, we’ll take a closer look at its definition and its calculation while providing examples of how it’s used in research. We’ll also discuss the importance of considering the margin of error when interpreting survey results and how it can affect the conclusions drawn from the data. So, whether you’re experienced or just starting your journey, this article is a must-read for anyone looking to master the art of margin of error and ensure the accuracy and reliability of their research. Let’s get started!

What is a Margin of Error?

Definition:

The margin of error in statistics is the degree of error in results received from random sampling surveys. A higher margin of error in statistics indicates less likelihood of relying on the results of a survey or poll , i.e. the confidence on the results will be lower to represent a population. It is a very vital tool in market research as it depicts the confidence level the researchers should have in the data obtained from surveys.

A confidence interval is the level of unpredictability with a specific statistic. Usually, it is used in association with the margin of errors to reveal the confidence a statistician has in judging whether the results of an online survey or online poll are worthy to represent the entire population.

A lower margin of error indicates higher confidence levels in the produced results.

When we select a representative sample to estimate full population, it will have some element of uncertainty. We need to infer the real statistic from the sample statistic. This means our estimate will be close to the actual figure. Considering margin of error further improves this estimate.

LEARN MORE: Population vs Sample

Margin of Error Calculation:

A well-defined population is a prerequisite for calculating the margin of error. In statistics, a “population” comprises of all the elements of a particular group that a researcher intends to study and collect data. This sampling error can be significantly high if the population is not defined or in cases where the sample selection process is not carried out properly.

Every time a researcher conducts a statistical survey, a margin of error calculation is required. The universal formula for a sample is the following:

p̂ = sample proportion (“P-hat”).

n = sample size

z = z-score corresponds to your desired confidence levels.

Are you feeling a bit confused? Don’t worry! you can use our margin of error calculator .

Example for margin of error calculation

For example, wine-tasting sessions conducted in vineyards depend on the quality and taste of the wines presented during the session. These wines represent the entire production and depending on how well the visitors receive them , the feedback from them is generalized to the entire production.

The wine tasting will be effective only when visitors do not have a pattern, i.e. they’re chosen randomly. Wine goes through a process to be palatable and similarly, the visitors also must go through a process to provide effective results.

The measurement components prove whether the wine bottles are worthy to represent the entire winery’s production or not. If a statistician states that the conducted survey will have a margin of error of plus or minus 5% at a 93% confidence interval. This means that if a survey was conducted 100 times with vineyard visitors, feedback received will be within a percentage division either higher or lower than the percentage that’s accounted 93 out of 100 times.

In this case, if 60 visitors report that the wines were extremely good. As the margin of error is plus or minus 5% in a confidence interval is 93%, in 100 visitors, it’s safe to conclude that the visitors who comment that the wines were “extremely good” will be 55 or 65 (93%) of the time.

To explain this further, let’s take an example of a survey on volunteering was sent to 1000 respondents out of which 500 agreed to the statement in the survey saying that volunteering makes life better. Calculate margin of error for 95% confidence level.

Step 1: Calculate P-hat by dividing the number of respondents who agreed with the statement in the survey to the total number of respondents. In this case, = 500/1000 = 50%

Step 2: Find z-score corresponding to 95% confidence level. In this case, z score is 1.96

Step 3: Calculate by putting these values in the formula

Step 4: Convert to a percentage

Margin of error in sample sizes:

In probability sampling , each member of a population has a probability of being selected to be a part of the sample. In this method, researchers and statisticians can select members from their area of research so that the margin of error in data received from these samples is as minimum as possible.

In non-probability sampling, samples are formed based on cost-effectiveness or convenience and not on the basis of application and because of this selection process, some sections of the population may get excluded. Surveys will be effective only on filtering members according to interests and application to the survey being conducted.

The industrial standard for confidence level is 95% and these are the margin of error percentages for certain survey sample sizes:

As indicated in this table, to reduce the margin of error to half, for instance, from 4 to 2, the sample size determination has been increased considerably, from 500 to 2000. As you must have observed, the sample size is inversely proportional to it. Till sample sizes of 1500, there is a significant decrease in it, but beyond that, this decrease reduces.

LEARN ABOUT: Survey Sampling

Ready to take your research game to the next level? Join QuestionPro and elevate your surveys from ho-hum to oh-wow! With our user-friendly platform, you’ll be able to create professional-grade surveys in no time. Plus, with features like customizable templates, advanced analytics, and real-time data collection, you’ll have everything you need to gather accurate, reliable data. So, don’t miss out on the fun. Join QuestionPro today for free and start discovering what your customers, employees, and communities really think. No credit card is needed.

LEARN ABOUT: Level of Analysis

LEARN MORE         SIGN UP FREE

MORE LIKE THIS

Raked Weighting: A Key Tool for Accurate Survey Results

May 31, 2024

Top 8 Data Trends to Understand the Future of Data

May 30, 2024

Top 12 Interactive Presentation Software to Engage Your User

May 29, 2024

Trend Report: Guide for Market Dynamics & Strategic Analysis

Other categories.

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Uncategorized
• Video Learning Series
• What’s Coming Up
• Workforce Intelligence

• Join Our Team
• Brand Positioning
• Competitor Activity
• Brand Awareness Surveys
• Brand Tracking Research
• Brand Sentiment Analysis
• Employee Engagement Surveys
• Culture Health Checks
• Social Research for the 3rd sector
• Brand Perception & Awareness
• User Needs Analysis
• Impact & Evaluation Research
• Customer Satisfaction Surveys
• Customer Perception Surveys
• Customer Experience Surveys
• Customer & Audience Segmentation
• Sentiment Analysis
• Mystery Shopping
• Usability Audit
• Net Promoter Score (NPS)
• B2B Market Research
• Bespoke Research
• Research Design
• Research Training Courses
• Qualitative Research
• Quantitative Research
• Secondary Research

Case Studies

• Market Research Testimonials
• News & Views

A Guide to Margin of Error, Confidence, and the Reliability vs Validity Debate

What is margin of error?  

When presenting statistics many larger organisations, such as Government departments and charities, like to understand the margin of error so that they feel confident that their findings are robust. The margin of error  informs the reader how accurately the results will represent the whole population, and when illustrated with a confidence level, will tell how often your margin of error will be accurate (the industry standard is 95%). In practice, a 95%  confidence level with a 6% margin of error means that your statistic will be within 6% points of the real population value, 95% of the time. In simple terms by calculating a confidence level and interval you can predict that your sample will yield results that are 95% (confidence level) representative of the mean (average) answer, with a + / –  6% range. Hence, the margin of error can be used to demonstrate the reliability of research findings. 

You may have seen predictive margin of error calculators elsewhere. These are calculators that ask you to supply your sample size and desired confidence level (and potentially your population size), and then provide you with a predictive margin of error for that research. For example, you may enter a population size of 100, a sample size of 80, and a desired confidence level of 95%, to which the calculator will happily tell you your margin of error is just 3%. However, if you head over to another margin of error calculator and enter the same details, it may tell you that your margin of error will be 10%.  

The difficulty is that margin of error cannot be calculated without knowing the results, as you need to be able to calculate the mean and standard deviation; this means that the predictive calculators are guessing your standard deviation to calculate the margin of error. Predictive margin of error calculators are estimations, and will not accurately represent the margin of error you will see from your survey results. 

Don’t get hung up on numbers (the reliability vs validity debate)  

If you are seeking a highly empirical, over-time comparison of changes in your customers' perceptions of your business, via benchmarking or Brand Tracking surveys, then ideally you will be looking to achieve the lowest feasible margin of error. However, applying a margin of error does not always deliver valuable insight. If your organisation's goal is to identify specific areas of improvement for your business, to increase customer loyalty and satisfaction, then you will find much greater benefit in conducting a deeper, qualitative survey with a smaller sample. In fact, some of the most influential ethnographic studies contain data relating to just 15 respondents. 

Long responses and open-ended questions will tell you far more about how you can improve your product or customer service, even if your sample is considerably smaller, than you could have achieved with a ‘catch-all’ NPS survey . Knowing, to a confidence level of 95% and a margin of error of 2%, that the vast majority of your customers think your communication could be improved, does not necessarily tell you how to improve your communication. If you instead discovered that 17 of 25 people surveyed liked to be communicated to via weekly emails, and explained which content would be helpful, you would be able to create targeted and engaging campaigns.  

Larger samples, with applied margins of error, help us to identify themes whilst smaller samples will reveal detailed information (why and how), yet it can be difficult to determine which approach is appropriate. Our expert team will advise on the most appropriate approach and will recommend a research method based upon your projects aims and objectives to ensure that your research project delivers the absolute best insight.

If this sounds like something your company might benefit from, you can contact us here .   

Research Executive

Jess has a Masters degree in Cybercrime Investigation, and a Bachelors in Sociology and Criminology. She loved the research and statistics aspects of her degrees and now enjoys experiencing the practical applications of research, alongside writing content and experimenting with new software. Her favourite part of research is finding meaningful answers hidden within data.

What Our Clients Say

• “Not only has the exercise helped us better understand the needs of older people in the county, but it also raised some important questions, which will help our organisation drive forward new ideas in the future.” Jonathan Skermer – Marketing/Development Manager, Age UK Suffolk
• “Their collaborative and plain speaking approach makes working with them a pleasure. High standards are maintained at every part of the research process from questionnaire design, through to data collection, analysis, and the concise presentation of results. In particular we appreciate their no-nonsense approach to reporting results in a concise and graphical manner that is easily understood.” Stephen Duffety – Senior Partner, Baker Tilly Tax and Accountancy
• “Mackman has worked professionally at all times, enabling clear presentation of the results to identify key trends and areas for growth and improvement of the Customer Services team now and in the future.” Sharlene Moffitt – International Customer Services Manager, Twinings

See how our insight makes a measurable difference.

Our latest Blog Posts

From research tips to industry updates.

Managing Stakeholder Conflict When Conducting Market Research

The Power of Customer-Centric Research & Human Understanding

How Can Employee Engagement Services Help Navigate The B Corp Certification Journey?

• Pollfish School
• Market Research
• Survey Guides
• Get started
• What is the Margin of Error & How to Reduce it in Your Survey

What is the Margin of Error & How to Reduce it in Your Survey

You may have pondered, what is the margin of error during your survey research, as this figure is sure to come up when analyzing data and forming an accurate reading of survey results.

In survey research and statistics, this pesky error rears its ugly head omnipresently. Commonly called the margin of error, this phenomenon is also referred to as the confidence interval. Despite the latter name, it is in the best interest of researchers to keep this metric low.

This is because, no matter who you survey or how many people, it is impossible to garner responses that represent everyone in the target population of your study. This is where the core of the margin of error lies, as it shows researchers how much of a survey’s results represent the views of its target population.

This article explains what is a margin of error and how to reduce it in your survey.

Defining the Margin of Error

The margin or error  — or the confidence interval — is a measurement of error in the results of a survey, specifically one that relies on the random sampling method . This metric shows researchers the degree to which they can expect survey results to reflect the views of the overall studied population.

As aforesaid, it is imperative to keep the margin of error low , as a high one points to a smaller likelihood of survey results to reflect the true views of your target population. As such, a higher margin of error renders your survey less reliable and inconclusive.

Moreover, a higher confidence interval (a misnomer as it may be) makes your survey unable to be used for statistical methods, such as in quantitative studies . It also debilitates descriptive and qualitative research . 

Researchers should thus aim for a lower margin of error, as it denotes higher confidence of a survey results’ accuracy, whereas a higher one signifies the opposite. 

The Utility of Studying the Margin of Error

While this statistic may be a frustrating reality in survey research, it is crucial to keep track of it. This is because survey data, necessary and enlightening as it may be, is imprecise. Why? Because survey samples represent a chunk (usually a small one) of your target population, and an even smaller one if it is in relation to your entire target market. 

Since a sampling pool (survey respondents) represents a larger population, there will always be the presence of uncertainty and imprecision. 

The confidence interval is useful in that it calculates this inherent imprecision to allow researchers to see how trustworthy their survey results are in representing a certain population.  

Additionally, the margin of error helps researchers determine the accuracy of a value, by expressing it in a range . The margin of error supports this, since the resulting answers of a survey question get assigned a percentage for statistical survey analysis. 

But since it is impossible to survey every last member of a target population, this percentage is only representative of the sampling pool . The margin of error allows this percentage to be expressed as a range to more accurately represent the answer in relation to the entire target population.

For example, if a survey’s results show that 40% of respondents sit while listening to music, this figure is incomplete. When you calculate the margin of error, on the other hand, it helps turn this figure into a more precise representation. If the margin of error turns out to be 4%, the percentage of 40% can be expressed as a range of 36-44%.

How to Calculate the Margin of Error

Calculating the margin of error requires plugging in a few variables into a formula. These include your sample size and the population standard deviation . The latter variable requires a calculation of its own. 

The standard deviation is a measure of the spread of scores in a data set that pertains to a specific population. A low standard deviation entails that most of the scores are closer to the average one, while a higher standard deviation shows that the scores are more dispersed. In short, this metric is also used for the purposes of data reliability.

These variables are just a few of the key aspects of calculating the margin of error. Below is the formula in its entirety.

The Formula Explained in Steps:

• Find the population standard deviation (σ) and determine your sample size (n).
• Get the square root of your sample size.
• Divide the population standard deviation (σ) by the result of the square root calculation.
• Multiply this result by the z-score that corresponds with your desired confidence level — not to be confused with the confidence interval (which is the margin of error). 

*Use the table below to find the z-score.  

How to Reduce the Margin of Error

As a researcher or business owner studying your target market, you are probably wondering how to reduce the margin of error. After all, a more reliable survey is going to be far more useful and one less obstacle to contend with. 

Here are several best practices to input into your survey research, to reduce the confidence interval. This will help make individual answers, as well as their ensuing variables (such as mean and average), become more precise. 

• Do this by changing how you gather the data. Ex: make precise measurements, make your process more orderly.
• Bear in mind that a larger sample size may require more time and resource allocation. 
• You should thus only lower the confidence level if the benefit of more precision (a reduced margin of error) outweighs the disadvantage of a lessened confidence level. 
• Example: If increasing the sample size is too costly for you, then lower the confidence level for a decreased confidence interval. Just remember, your confidence level has declined.
• Implement a one-sided confidence interval. This has a smaller margin of error than a two-sided confidence interval. This interval signifies only if a parameter is either smaller than or greater than a cut-off value. 
• It doesn’t show any information about the parameter on the opposite side. As such, use this to increase an estimate’s precision of an estimate only if you fear the estimate will be greater or less than a cut-off value, i.e., not both.

Using the Correct Survey Platform

Can using a strong online survey tool help decrease the margin of error? The answer is rather complicated. On one hand, a strong survey tool will help you avoid inaccuracies (think gibberish answers, straightlining, etc.) and illogical questions. It can also help ambiguity with open-ended questions.

These aspects are certainly crucial in helping eliminate ambiguity. But there are things that affect the margin of error that a survey simply has no control over.

For example, the population size and confidence level are factors outside of a survey and its hosting platform. 

With this in mind, you should understand that the margin of error cannot be fully diminished. It is impossible to perfectly align with the population you’re surveying. Especially when certain respondents have a tendency to change their minds. However, you can still determine how close you are to precisely reflecting their views by determining the margin of error yourself, or by calculating it with a designated tool (such as the one hyperlinked at the very beginning of this article).

What’s most important about an online survey platform is that it can deliver peace of mind on so many aspects that go into survey research , minimizing the labor that you need to exert into your studies and making the process much smoother and easier when it comes to reaching results (and analyzing them). 

Do you want to distribute your survey? Pollfish offers you access to millions of targeted consumers to get survey responses from $0.95 per complete. Launch your survey today.

Privacy Preference Center

Privacy preferences.

Sample Size Calculator and Margin of Error: The Ultimate Guide by Appinio

Appinio Research · 20.12.2022 · 19min read

Whether it's about political sentiment, customer satisfaction, the Appinio Hype Train or a company employee survey, surveys are the most common way to find out public sentiment, opinions, beliefs and tendencies. And of course the more people you ask questions to, the closer you get to a representative result. But imagine interviewing all the people in the UK to find out who they would vote for in the next election, impossible!

This is where the need of a representative sample comes into play.

A representative sample focuses on a portion of the population and aims to reflect the views of the general public. But, especially in survey research, what variables do you need to take into account? How do you know how many survey respondents you require to ensure you survey a representative sample of the population? In this article, we’ll show you which key values you need in order to calculate the perfect sample for your study and how accurately your results represent the population.

Do you want to calculate the sample size for your study quickly and already have all the key values? Then use our online sample size calculator margin or error to determine the sample you need.

Sample size calculator

Use our calculator to quickly check the sample size for your study. Just enter your values and check the result. All metrics used in the calculator are explained in detail later in the article. If you have more questions, feel free to reach out to

our experts via the chat.

Definition: What is a sample?

A sample is a small portion or quantity intended to show what the whole is like.

There are several different methods of sampling that can be used, for example, you could choose to use a “random sample,” where respondents are chosen entirely by chance from the population at large, therefore every member of the population has an equal chance to be selected for your study.

A random sample is useful when doing exploratory market research or gathering new ideas, but it would not be useful if you wanted to know how many Brits are vegan, as your random sample could exclude some participants (e.g. not enough women) or include too many people from the same group (e.g. too many Millennials), therefore, you would not be able to extract insights that are applicable to your desired target group or the general population.

The reason why sample size is important

That's why, in survey research especially, a sample needs to meet specific criteria and contain all the different groups - such as women, men or different age groups - in the same proportions to the population you want to represent. In addition, the sample should be large enough not only to be representative, but also to make reliable statements. When choosing a sample it is important to test its reliability, if a sample is too small the information collected may be partial, on the other hand if the sample is too large it would be a waste of resources. The larger the group, the closer are results to the target market, the less chance factors, errors and false statements can distort the result. The results are considered representative of the whole itself, i.e. applicable to the general public, when they reflect the normal distribution in the population

If you want to get survey data that is as reliable as possible, you need to strive to make your survey representative. In market research, you need to collect data from the target group before launching a product on the market in order to offer a product that consumers will want to buy or use. However, you can't ask every potential buyer their opinion, it would simply be too time-consuming and not cost-effective. Moreover, it is impossible to determine exactly the size of the potential buyer group. Therefore, before sending your survey out to your respondents, you need to define a sample size that represents your target population as closely as possible, and in all its facets.

How large should a sample be?

The larger the sample size, the more accurate results are, right?

Generally, the larger the sample, the more representative it is, that is under the condition that we are surveying the right people; namely, we won't survey men when we want to represent pregnant women.

The results of a survey are considered representative when the outcomes of the survey accurately reflect the overall population which means that you can draw reliable conclusions about the general population as all characteristics of the target population are also present in your sample.

So a larger sample makes reaching representativeness easier, but on the other hand, the larger the sample size, the more expensive and time-consuming your survey will be. So, how can you balance costs and representativeness?

In order to do this, you must make trade-offs between the size of the sample and the margin of error, namely, you have to find a sample size that is large enough to achieve the desired level of precision and stay within an acceptable margin of error.

Let’s have a look at the margin of error metric and what margin of error is considered acceptable.

What is the margin of error?

The margin of error (or marginal error) is a statistic expressing the amount of random sampling error in the results of a survey. The larger the margin of error, the less confidence one should have that a poll result would precisely reflect the result of a census of the entire population. The margin of error relates to the size of your sample and the difference between results from your survey data and those from population (normal) distribution, it tells you how close your results reflect the views from the overall population. Let's make an example.

As you can guess by the word margin, the margin of error is a range of values around the sample mean. For example, a survey of 1000 people of your targeted population gives you a 5% margin of error, meaning that 95% of the time the results are within 5% range of what they would be if you had asked every single person in your target population. Thus, if your survey data indicates that 40% of the total population approve or disapprove of something, the margin of error means that you can be 95% confident that the true figure falls between 35% and 45%. The marginal error indicates how close the results of the sample are to reality. This can be calculated by taking into account the sample size, margin of error and confidence level. The marginal error should not be too high, otherwise it can lead to wrong conclusions which could have serious consequences. An acceptable margin of error used by most survey researchers typically falls between 4% and 8% at the 95% confidence level. It is important to keep the margin of error as low as possible as the smaller the margin of error, the more confidence you may have in your results, the bigger the margin of error, the farther they can stray from the views of the total population, making your results unusable. As a rule of thumb, as sample size grows marginal error decreases but, it should be noted that a too large sample size makes the survey more expensive and time consuming. Therefore, finding a balance between marginal error and sample size is essential. With proper planning you can make sure that your sample is representative of the target population, with marginal errors as low as possible so your results are accurate and reliable.  Let’s keep in mind that what is considered an acceptable value also depends on the kind of research you’re conducting. For obvious reasons, medical studies and trials have higher standards and stricter criteria when it comes to sampling and collecting results than researching consumers' purchasing behaviour.

Margin of error calculator

Use our calculator to quickly check the margin of error for your study. Just enter your values and check the result. All metrics used in the calculator are explained in detail below. If you have more questions, feel free to reach out to our experts via the chat.

An acceptable margin of error used by most researchers typically falls between 3% and 6% at the 95% confidence level.

The key values to calculate sample size and margin of error

The key values to determine the optimal sample size and / or margin of error are the total population N, the standard deviation p, confidence interval (CI) and the Z-value z. Here is a brief explanation for every metric.

Standard deviation, p

When looking at survey results, you should be able to make general statements such as "xx% approve of this". However, this can vary greatly between respondents, which is why we also need to take into account standard deviation. Standard deviation is expressed in percentage and specifies the range of probable answers for each question. The higher this value, the more participants are needed to obtain an accurate result. The standard deviation value is usually set at 0.5 and indicates how much variation is expected in the answers of the respondents. Here's an example of how a high standard deviation can occur when using a Likert scale question: Imagine a survey is conducted to measure people's opinion on a new product. One of the questions is: How likely are you to recommend this product to a friend? Respondents reply via a Likert scale question with five options: "Strongly unlikely", "Unlikely", "Neutral", "Likely", "Strongly likely". Out of 100 respondents, 50 of them choose "Strongly unlikely" and the other 50 choose "Strongly likely". This means there is no middle ground in the responses, they are all at one of the two extremes. As we have half of the respondents choosing one extreme and the other half choosing the other extreme, the responses are not evenly distributed and the standard deviation would be much higher to indicate the large difference or spread between the responses. This high standard deviation is unusual when compared to a "normal" distribution of responses, where you would expect most responses to be close to the average, with fewer responses at the extremes. It is important to notice that a standard deviation value of 0.5 is considered a common value and typically only needs to be adjusted in edge cases, like the one mentioned in this example. If you have a specific use case in mind and are unsure on what kind of standard deviation to expect, you can also contact our experts for support.

Get a free demo and see the Appinio platform in action!

Z-score, z, and Confidence interval, (CI)

The Z-score measures how well the sample represents the total population (including the margin of error) and describes the deviation of a value from the average of your sample. Or in other words: The Z-value indicates how certain you could be that your study results correspond to reality. 

As it is more tangible, confidence intervals that represent a certain z-score are usually used for calculating a desired sample size or margin of error. This table shows the z-scores for the most common confidence intervals.

A confidence interval is a range of values, derived from a sample, that is used to estimate an unknown population parameter. It provides a level of certainty or confidence that the true population parameter falls within the interval.  The interval is calculated by taking a sample statistic (such as the mean or proportion) and adding and subtracting a margin of error, which is determined by the level of confidence desired and the sample size. The level of confidence is typically expressed as a percentage, such as 90%, 95%, or 99%.

The confidence interval is the range of values you expect your estimate to fall between if you redo your test, within a certain level of confidence.

Let’s use a visual example using a normal distribution.

For example, the probability of the population mean value being between -1.96 and +1.96 standard deviations (z-scores) from the sample mean is 95%.

Accordingly, there is a 5% chance that the population mean lies outside of the upper and lower confidence interval (as illustrated by the 2.5% of outliers on either side of the 1.96 z-scores).

Total population, N

In most everyday uses, the word population implies a group of people or at least a group of living beings. However, statisticians and researchers refer to whatever group they are studying as a population . The population of a study might be mothers of children under the age of 5 years old, doctors or users of a specific product. In order to draw the most precise conclusions possible, statisticians and researchers would need to know all the characteristics of the people in the desired population, but this is impossible or impractical most of the time, since population sets tend to be quite large. That’s why they select samples of the population, namely a smaller group drawn from the total population that has the characteristics of the entire population, so that the observations and conclusions made against the sample data can be attributed to the population as a whole. In our Appinio calculator we do not consider the total population as for most studies the population size is so large (e.g. all Germans) that it doesn’t affect the required sample size or margin of error. Only when the population you want to investigate is extremely small - like all male dentists in London - the total population needs to be taken into account. In this case you can check out our advanced formula below or directly consult our experts for help.

Open live chat

The case of unknown population

In most cases, determining the population size precisely is not possible, because it is too large and broad. In these cases, you can only use the upper part of the standard formula to calculate the minimum sample size:

Formula for determining the sample size

For completion’s sake, we are also going to show you how you can calculate the appropriate sample size using the extended formula, but if you are not a fan of maths and formulas bring back school memories that you’d like to keep hidden, use our Appinio calculators above. Once you determine all key values mentioned above, you can use the equation below to determine the optimal sample size. The following standard formula is best suited for small to medium population sizes.

Conclusions

In conclusion, a reliable sample size is critical for making accurate, representative and meaningful conclusions when conducting survey research. This ensures that marginal errors are minimised and that results can be extrapolated from the population with greater certainty. By following the outlined formula, you can determine the optimal sample size for your study to ensure accurate and reliable results.

We have discussed marginal error, sample size and the calculation of Z-value. To determine the sample size for a survey, you must use the standard formula and consider three key values: Margin of error, standard deviation, and Z-value. In specific use cases where you want to represent a very small population, population size needs to be considered additionally.  By following this formula, you can ensure accuracy and reliability in your survey results. The marginal error is a measure used to describe how well a sample represents the total population and refers to the likelihood of an error occurring. The Z-value is calculated based on the confidence level and indicates how certain you can be that your results correspond to reality. Finally, the sample size should be determined using the standard formula to minimise marginal errors and extrapolate results to the full population accurately.

Do you want to start your own survey but don't quite know how?

Have a look at the Appinio Hype Tracker Report! For the Appinio Hype Tracker Reports we investigate a population that is representative according to age and gender. Find all Appinio Hype Tracker volumes on our Report Page and register for the dashboard free of charge here.

Get free access to the platform!

Join the loop 💌

Be the first to hear about new updates, product news, and data insights. We'll send it all straight to your inbox.

Get the latest market research news straight to your inbox! 💌

Wait, there's more

30.05.2024 | 29min read

Pareto Analysis: Definition, Pareto Chart, Examples

28.05.2024 | 32min read

What is Systematic Sampling? Definition, Types, Examples

16.05.2024 | 30min read

Time Series Analysis: Definition, Types, Techniques, Examples

Guide to margin of error (with examples)

Last updated

4 March 2023

Reviewed by

Let's dive deeper to understand the margin of error and how to handle it. Get ready to crunch some numbers.

Free template to analyze your survey results

Analyze your survey results in a way that's easy to digest for your clients, colleagues or users.

• What is the margin of error in a survey?

The margin of error is the range within which the actual value of a survey parameter falls within a certain confidence level. In other words, it quantifies how the results from a sample might differ from the actual value you would have obtained if you had studied the whole population.

The confidence interval (CI) is the range between the upper and lower bounds of an estimate. The narrower the interval, the better and more accurate your results.

For instance, suppose a random sample of people has a +/−2% margin of error at a 95% confidence level. That implies that if you repeated the same survey 100 times, you expect the percentage of people who gave a particular answer to fall within the range of 2% of the reported results, 95 times.

The margin of error usually decreases with an increase in the random sample size or the number of responses. That means you can be more confident your results are reliable.

• When can you use a margin of error?

You use a margin of error when you have a random sample—a set of randomly selected respondents from the whole population you’re studying. A random sample is also known as a probability sample since every member of your population has a known probability of being part of your sample.

For example, your company wants to know whether workers would prefer an extra leave day or bonus pay. You can randomly select a few employees from your workforce and ask them to choose their preferences. A margin of error will help the decision-makers understand the accuracy of the results.

• What factors affect the margin of error?

The size of the margin of error depends on various factors:

Sample size: the more respondents who complete your study, the smaller the margin of error

Confidence level (CL): increasing the confidence level leads to a wider margin of error

Population variance: the higher the variance, the larger the margin of error

Poll design: your questions' exact wording can influence how people answer them, affecting the MOE

Response rate : a lower response rate increases the MOE. It can also increase if the respondents don't closely resemble the larger population.

Non-sampling errors: errors from sources such as coding and measurement can affect the margin of error

• What is an acceptable margin of error for a quantitative survey?

The MOE size varies with the percentage of the target population sampled. Knowing the MOE for a particular study is crucial since the value for all percentages will be assessed through this lens when reviewing data.

If you were surveying 1,000 people using a confidence level of 90, surveying at least 250 people would give you an MOE of 4%. With this margin, you could be reasonably confident that the results were reflective of the audience you were surveying.

• How to calculate the margin of error

Whenever you perform a statistical survey, you should calculate the MOE. Before you do so, you need to define your population.

A population is a group of elements from which you want to survey and gather data. Properly defining the population and selecting an appropriate sample size can help reduce the MOE.

Margin of error calculator .css-5oqtrw{background:transparent;border:0;color:#0C0020;cursor:pointer;display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;font-size:18px;font-weight:600;line-height:40px;outline:0;padding:0;} .css-17ofuq7{-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;background:transparent;border:0;color:inherit;cursor:pointer;-webkit-flex-shrink:0;-ms-flex-negative:0;flex-shrink:0;background:transparent;border:0;color:#0C0020;cursor:pointer;display:-webkit-inline-box;display:-webkit-inline-flex;display:-ms-inline-flexbox;display:inline-flex;font-size:18px;font-weight:600;line-height:40px;outline:0;padding:0;}.css-17ofuq7:disabled{opacity:0.6;pointer-events:none;} .css-7jswzl{-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;display:inline-block;height:28px;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;width:28px;-webkit-text-decoration:none;text-decoration:none;}.css-7jswzl svg{height:100%;width:100%;margin-bottom:-4px;}

Optimize your research’s impact when you improve the margin of error.

Margin of error

The total number of people whose opinion or behavior your sample will represent.

The probability that your sample accurately reflects the attitudes of your population. The industry standard is 95%.

The number of people who took your survey.

Here's the formula:

MOE= z-value √ (p(1-p)/n)

n is the sample size

p is the sample percentage

z-value is the critical value corresponding to your confidence level

Here are standard confidence levels with their corresponding z-values:

Finding the maximum margin of error formula

For the maximum MOE, you'll use 0.5 as p and the z-value for 95% (the standard confidence level), which is 1.96. Thus, the formula is a direct transformation of the sample size:

Maximum 95% MOE = 1.96√0.5 (1 - 0.5)/n =0.98/√n

Plug in your sample size to the formula, and voila! You'll get the maximum margin of error for whatever survey you've done. That means the figure is unique for a given size.

For example, the maximum MOE for a sample size of 100 would be 0.98/√100 = 0.098

How to calculate the margin of error with your survey data

Now you have your formula, crunching those figures won't be a headache.

Take n , the sample size, and p , the sample percentage.

Calculate p(1 - p) and divide the result by n .

Find the square root of the value.

Multiply the figure you got in Step 3 by the z-value .

An example of calculating the margin of error

Suppose you surveyed 1,000 respondents to find out their views on volunteering. 500 of them agreed that volunteering is an excellent part of life. What's the margin of error given a 50% CL?

MOE= z√(p (1 - p))/n

P = 500/1,000 = 50%

z-score = 1.96 (at 95% confidence level)

Therefore, MOE= 1.96√ (0.5(1 - 0.5))/1,000

=0.031=3.1%

So the sample's MOE is +/−3%. That implies if you repeated the survey several times, the number of people who love volunteering would be within 3% of the sample percentage (50%) 95% of the time.

• How sample size affects the margin of error

The size of the sample is a key factor that affects your margin of error. Looking to increase your study’s precision? Interview more respondents and ensure they complete your survey.

Increasing sample size reduces the standard deviation (a measure of your estimate's variability). In other words, it narrows the range of possible values, leading to more precise estimates and lower MOE.

For example, if you're looking to estimate the average income of your existing customers, a sample of two people will lead to a wide range of figures, so the margin of error will be high. But if you increase the sample size to 1,000 people, the MOE will be significantly narrower.

That means MOE is critical to understanding whether your sample size is appropriate. Is it too large? You'll need to survey more people to capture your population's attitudes accurately.

Calculate your ideal sample size

• How to increase your data's reliability

If you're looking to boost your survey's reliability, the trick is to minimize the MOE. Here are three tried-and-true ways you can generate more accurate results:

1. Minimize the variables

A high number of variables can introduce more errors to your survey. Variables cause your standard variation to shoot up, increasing the MOE. Here's a trick: change how you collect data. For example, ensure the process is rigorous and measure your factors or variables accurately.

2. Increase the sample size

This hack is often the easiest on our list. Statistically, if more people complete your study, your chances of getting a representative response increase because the confidence interval decreases. The result is a lower MOE. But ensure you have enough resources and time to generate a larger sample size.

3. Lower your confidence level

Using lower confidence is another trick that can lead to a narrower MOE. But be careful since a lower level means decision-makers will be less confident in the results.

Therefore, only reduce the CL if the cons of a smaller MOE outweigh the cons of a lower CL. For example, if it’s unfeasible to increase the sample size due to costs, you can reduce the CL to achieve a narrower interval.

How can you calculate the margin of error when you have a confidence level?

You can calculate the margin of error using the following formula:

MOE= z √(p (1-p))/n

Where z is the z-value for your desired confidence level, p is the sample percentage, and n is the sample size.

What is a large margin of error?

A large MOE implies a high chance of the actual value being very different from the value you've estimated. It often occurs when dealing with small sample sizes and high data variability, combined with a high CL.

What if the margin of error is very high?

If the margin of error is very high, the survey's sample results won't accurately represent your population, so decision-makers can't rely on it. The solution is to survey more people, reduce variables, lower the CL, or use a one-sided CI.

Should you be using a customer insights hub?

Do you want to discover previous survey findings faster?

Do you share your survey findings with others?

Do you analyze survey data?

Start for free today, add your research, and get to key insights faster

Editor’s picks

Last updated: 4 March 2023

Last updated: 20 March 2024

Last updated: 22 February 2024

Last updated: 12 May 2023

Last updated: 23 May 2023

Last updated: 21 December 2023

Last updated: 14 February 2024

Last updated: 11 March 2023

Last updated: 13 May 2024

Last updated: 30 January 2024

Latest articles

Related topics, .css-je19u9{-webkit-align-items:flex-end;-webkit-box-align:flex-end;-ms-flex-align:flex-end;align-items:flex-end;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-box-flex-wrap:wrap;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;row-gap:0;text-align:center;max-width:671px;}@media (max-width: 1079px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}}@media (max-width: 799px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}} decide what to .css-1kiodld{max-height:56px;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}@media (max-width: 1079px){.css-1kiodld{display:none;}} build next, decide what to build next.

Users report unexpectedly high data usage, especially during streaming sessions.

Users find it hard to navigate from the home page to relevant playlists in the app.

It would be great to have a sleep timer feature, especially for bedtime listening.

I need better filters to find the songs or artists I’m looking for.

Log in or sign up

Get started for free

Margin of Error: Connecting Chance to Plausible

• First Online: 15 June 2023

Cite this chapter

• Gail Burrill   ORCID: orcid.org/0000-0001-9629-4100 11  

Part of the book series: Advances in Mathematics Education ((AME))

274 Accesses

An intriguing question in statistics: how is it possible to know about a population using only information from a sample? This paper describes a simulation-based formula-light approach used to answer the question in a course for elementary preservice teachers. Applet-like dynamically linked documents allowed students to build “movie clips” of the features of key statistical ideas to support their developing understanding. Live visualizations of simulated sampling distributions of “chance” behavior can enable students to see that patterns of variation in the aggregate are quite predictable and can be used to reason about what might be likely or “plausible” for a given sample statistic. The discussion includes an overview of a learning trajectory leading to understanding margin of error, summarizes a key lesson in the development, describes a brief analysis of student understanding after the course, and concludes with some recommendations for future investigation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Advanced Placement is a program that allows high school students to receive college credit for successfully taking a college course in high school.

Bakker, A. (2004). Reasoning about shape as a pattern in variability. Statistics Education Research Journal, 3 (2), 64–83. https://doi.org/10.52041/serj.v3i2.552

Article   Google Scholar  

Bakker, A. (2018). Design research in education. In A practical guide for early career researchers . Routledge. https://doi.org/10.4324/9780203701010

Chapter   Google Scholar  

Ben-Zvi, D., Bakker, A., & Makar, K. (2015). Learning to reason from samples. Educational Studies in Mathematics, 88 (3), 291–303. https://doi.org/10.1007/s10649-015-9593-3

Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The Solo taxonomy . Academic.

Google Scholar  

Bransford, J. D., & Brown, A. L. (1999). In R. R. Cocking (Ed.), How people learn: Brain, mind, experience, and school . National Academy Press.

Budgett, S., & Rose, D. (2017). Developing statistical literacy in the final school year. Statistics Education Research Journal, 16 (1), 139–162. https://doi.org/10.52041/serj.v16i1.221

Burrill, G. (2018). Concept images and statistical thinking: The role of interactive dynamic technology. In M. A. Sorto (Ed.). Proceedings of the tenth international congress on teaching statistics . Kyoto, Japan. https://iaseweb.org/Conference_Proceedings.php?p=ICOTS_10_2018

Castro, A., Vanhoof, S., Van den Noortgate, W., & Onghena, P. (2007). Students’ misconceptions of statistical inference: A review of the empirical evidence from research on statistics education. Educational Research Review, 2 (2), 98–113. https://doi.org/10.1016/j.edurev.2007.04.001

Chance, B., & Rossman, A. (2006). Using simulation to teach and learn statistics. In A. Rossman & B. Chance (Eds.), Working cooperatively in statistics education: Proceedings of the seventh international conference on teaching statistics (pp. 1–6). Salvador. https://iase-web.org/Conference_Proceedings.php?p=ICOTS_7_2006

Cobb, G. (2007). The introductory statistics course: A Ptolemaic curriculum? Technology Innovations in Statistics Education, 1 (1), 1–15. https://doi.org/10.5070/T511000028

Crooks, N., Bartel, A., & Albali, M. (2019). Conceptual knowledge of confidence intervals in psychology undergraduate and graduate students. Statistics Education Research Journal, 18 (1), 46–62. https://doi.org/10.52041/serj.v18i1.149

del Mas, R., Garfield, J., & Chance, B. (1999). A model of classroom research in action: Developing simulation activities to improve students’ statistical reasoning. Journal of Statistics Education, 7 (3). https://doi.org/10.1080/10691898.1999.12131279

del Mas, R., Garfield, J., Ooms, A., & Chance, B. (2007). Assessing students’ conceptual understanding after a first year course in statistics. Statistics Education Research Journal, 6 (2), 28–58. https://doi.org/10.52041/serj.v6i2.483

Drijvers, P. (2015). Digital technology in mathematics education: Why it works (or doesn’t). In S. J. Cho (Ed.), Selected regular lectures from the 12th international congress on mathematical education (pp. 135–151). Springer. https://doi.org/10.1007/978-3-319-17187-6_8

Fidler, F. (2006). Should psychology abandon p -values and teach CIs instead? Evidence-based reforms in statistics education. In A. Rossman & B. Chance (Eds.), Working cooperatively in statistics education: Proceedings of the seventh international conference on teaching statistics (ICOTS-7) . Salvador. https://iase-web.org/documents/papers/icots7/5E4_FIDL.pdf

Fidler, F., & Loftus, G. R. (2009). Why figures with error bars should replace p -values: Some conceptual arguments and empirical demonstrations. Zeitschrift für Psychologie/Journal of Psychology, 217 (1), 27–37.

GAISE College Report ASA Revision Committee. (2016). Guidelines for assessment and instruction in statistics education college report . http://www.amstat.org/education/gaise

García-Pérez, M. A., & Alcalá-Quintana, R. (2016). The interpretation of scholars’ interpretations of confidence intervals: Criticism, replication, and extension of Hoekstra et al. (2014). Frontiers in Psychology, 7 , 1–12. https://doi.org/10.3389/fpsyg.2016.01042

Gnanadesikan, M., & Scheaffer, R. (1987). The art and technique of simulation. Pearson Learning.

Grant, T., & Nathan, M. (2008). Students’ conceptual metaphors influence their statistical reasoning about confidence intervals . WCER Working Paper No. 2008-5. Wisconsin: Wisconsin Center for Education Research. [Online: https://wcer.wisc.edu/docs/working-papers/Working_Paper_No_2008_05.pdf ]

Henriques, A. (2016). Students’ difficulties in understanding of confidence intervals. In D. Ben-Zvi & K. Makar (Eds.), The teaching and learning of statistics (pp. 129–138). Springer. https://doi.org/10.1007/978-3-319-23470-0_18

Hoekstra, R., Kiers, H., & Johnson, A. (2012). Are assumptions of well-known statistical techniques checked, and why (not)? Frontiers in Psychology, 3 . https://doi.org/10.3389/fpsyg.2012.00137

Hoekstra, R., Morey, R., Rouder, J., & Wagenmakers, E. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21 (5), 1157–1164. https://doi.org/10.3758/s13423-013-0572-3

Kalinowski, P., Lai, J., & Cumming, G. (2018). A cross-sectional analysis of students' intuitions when interpreting CIs. Frontiers in Psychology, 9 (112). https://doi.org/10.3389/fpsyg.2018.00112

Landwehr, J., Swift, J., & Watkins, A. (1987). Exploring surveys and information from samples . Pearson Learning.

Lane, D., & Peres, S. (2006). Interactive simulations in the teaching of statistics: Promise and pitfalls. In A. Rossman & B. Chance (Eds.), Working cooperatively in statistics education: Proceedings of the seventh international conference on the teaching of statistics . Salvador. https://iase-web.org/Conference_Proceedings.php?p=ICOTS_7_2006

Lipson, K. (2002). The role of computer-based technology in developing understanding of the concept of sampling distribution. In Proceedings of the 6th International Conference on Teaching Statistics .

Liu, Y., & Thompson, P. W. (2009). Mathematics teachers’ understandings of proto-hypothesis testing. Pedagogies, 4 (2), 126–138. https://doi.org/10.1080/15544800902741564

Makar, K., & Rubin, A. (2009). A framework for thinking about informal statistical inference. Statistics Education Research Journal, 8 (1), 82–105. https://doi.org/10.52041/serj.v8i1.457

Michigan Department of Education. (2016). M-STEP Final Reports Webcast .

Oehrtman, M. (2008). Layers of abstraction: Theory and design for the instruction of limit concepts. In M. Carlson & C. Rasmussen (Eds.), Making the connection: Research and teaching in undergraduate mathematics education . http://hub.mspnet.org//index.cfm/19688

Pfannkuch, M. (2008). Building sampling concepts for statistical inference: A case study. In Proceedings of the eleventh international congress of mathematics education (ICOTS 11) . Monterrey, Mexico. Online: http://tsg.icme11.org/tsg/show/15

Pfannkuch, M., & Budgett, S. (2014). Constructing inferential concepts through bootstrap and randomization-test simulations: A case study. In K. Makar, B. de Sousa, & R. Gould (Eds.). Sustainability in statistics education. Proceedings of the ninth international conference on teaching statistics (ICOTS9) Flagstaff, Arizona, USA.

Pfannkuch, M., Arnold, P., & Wild, C. (2015). What I see is not quite the way it really is: Students’ emergent reasoning about sampling variability. Educational Studies in Mathematics., 88 , 343–360. https://doi.org/10.1007/s10649-014-9539-1

Reading, C., & Reid, J. (2006). An emerging hierarchy of reasoning about distribution from a variation perspective. Statistics Education Research Journal, 5 (2), 46–68. https://doi.org/10.52041/serj.v5i2.500

Rumsey, D. (2022). Statistics II. For dummies (2nd ed.). John Wiley & Sons.

Sacristan, A., Calder, N., Rojano, T., Santos-Trigo, M., Friedlander, A., & Meissner, H. (2010). The influence and shaping of digital technologies on the learning – And learning trajectories - of mathematical concepts. In C. Hoyles & J. Lagrange (Eds.), Mathematics education and technology - rethinking the terrain (The 17th ICMI Study) (pp. 179–226). Springer. https://doi.org/10.1007/978-1-4419-0146-0_9

Saldanha, L. A. (2003). “Is this sample unusual?” An investigation of students exploring connections between sampling distributions and statistical inference . Unpublished doctoral dissertation, Vanderbilt University, Nashville, TN.

Saldanha, L., & Thompson, P. (2014). Conceptual issues in understanding the inner logic of statistical inference: Insights from two teaching experiments. Journal of Mathematical Behavior, 35 , 1–30. https://doi.org/10.1016/j.jmathb.2014.03.001

Scheaffer, R., Gnanadesikan, M., Watkins, A., & Witmer, J. (1996). Activity-based statistics . Springer.

Book   Google Scholar  

Shaughnessy, J. M. (2007). Research on statistics learning and reasoning. In F. K. Lester Jr. (Ed.), Second handbook of research on mathematics teaching and learning, 2 (pp. 957–1009). Information Age.

Stat Trek. Statistics dictionary. Teach yourself statistics. Accessed 6/30/2019. https://stattrek.com/statistics/dictionary.aspx?definition=margin%20of%20error

Tall, D., & Vinner, S. (1981). Concept image and concept definition in mathematics with particular reference to limits and continuity. Educational Studies in Mathematics, 12 , 151–169. https://doi.org/10.1007/BF00305619

Thompson, P. W., & Liu, Y. (2005). Understandings of margin of error. In S. Wilson (Ed.), Proceedings of the twenty-seventh annual meeting of the International Group for the Psychology of mathematics education . Virginia Tech.

Thornton, R., & Thornton, J. (2004). Erring on the margin of error. Southern Economic Journal, 71 (1), 130–135. https://doi.org/10.1002/j.2325-8012.2004.tb00628.x

Tintle, N., Carver, R., Chance, B., Cobb, G., Rossman, A., Roy, S., Swanson, T., & Vander Stoep, J. (2019). Introduction to statistical investigations . Wiley.

Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76 , 105–110. https://doi.org/10.1037/h0031322

Van Dijke-Droogers, M. J. S., Drijvers, P. H. M., & Bakker, A. (2020). Repeated sampling with a black box to make informal statistical inference accessible. Mathematical Thinking and Learning, 22 (2), 116–138. https://doi.org/10.1080/10986065.2019.1617025

Watkins, A., Bargagliotti, A., & Franklin, C. (2014). Simulation of the sampling distribution of the mean can mislead. Journal of Statistics Education, 22 (3), 1–21. https://doi.org/10.1080/10691898.2014.11889716

Wild, C. J., Pfannkuch, M., Regan, M., & Horton, N. J. (2011). Towards more accessible conceptions of statistical inference. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174 (2), 247–295. https://doi.org/10.1111/j.1467-985X.2010.00678.x

Download references

Author information

Authors and affiliations.

Program in Mathematics Education, Michigan State University, East Lansing, MI, USA

Gail Burrill

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Gail Burrill .

Editor information

Editors and affiliations.

Michigan State University, East Lansing, MI, USA

Gail F. Burrill

Federal University of Uberlândia, Ituiutaba, Brazil

Leandro de Oliveria Souza

University of San Carlos, Cebu City, Philippines

Enriqueta Reston

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Burrill, G. (2023). Margin of Error: Connecting Chance to Plausible. In: Burrill, G.F., de Oliveria Souza, L., Reston, E. (eds) Research on Reasoning with Data and Statistical Thinking: International Perspectives. Advances in Mathematics Education. Springer, Cham. https://doi.org/10.1007/978-3-031-29459-4_15

Download citation

DOI : https://doi.org/10.1007/978-3-031-29459-4_15

Published : 15 June 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-29458-7

Online ISBN : 978-3-031-29459-4

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

• Publications
• Our Methods
• Short Reads
• Tools & Resources

Read Our Research On:

5 key things to know about the margin of error in election polls

In presidential elections, even the smallest changes in horse-race poll results seem to become imbued with deep meaning. But they are often overstated. Pollsters disclose a margin of error so that consumers can have an understanding of how much precision they can reasonably expect. But cool-headed reporting on polls is harder than it looks, because some of the better-known statistical rules of thumb that a smart consumer might think apply are more nuanced than they seem. In other words, as is so often true in life, it’s complicated.

Here are some tips on how to think about a poll’s margin of error and what it means for the different kinds of things we often try to learn from survey data.

What is the margin of error anyway?

Because surveys only talk to a sample of the population, we know that the result probably won’t exactly match the “true” result that we would get if we interviewed everyone in the population. The margin of sampling error describes how close we can reasonably expect a survey result to fall relative to the true population value. A margin of error of plus or minus 3 percentage points at the 95% confidence level means that if we fielded the same survey 100 times, we would expect the result to be within 3 percentage points of the true population value 95 of those times.

The margin of error that pollsters customarily report describes the amount of variability we can expect around an individual candidate’s level of support. For example, in the accompanying graphic, a hypothetical Poll A shows the Republican candidate with 48% support. A plus or minus 3 percentage point margin of error would mean that 48% Republican support is within the range of what we would expect if the true level of support in the full population lies somewhere 3 points in either direction – i.e., between 45% and 51%. 

How do I know if a candidate’s lead is ‘outside the margin of error’?

News reports about polling will often say that a candidate’s lead is “outside the margin of error” to indicate that a candidate’s lead is greater than what we would expect from sampling error, or that a race is “a statistical tie” if it’s too close to call. It is not enough for one candidate to be ahead by more than the margin of error that is reported for individual candidates (i.e., ahead by more than 3 points, in our example). To determine whether or not the race is too close to call, we need to calculate a new margin of error for the difference between the two candidates’ levels of support. The size of this margin is generally about twice that of the margin for an individual candidate. The larger margin of error is due to the fact that if the Republican share is too high by chance, it follows that the Democratic share is likely too low, and vice versa.

For Poll A, the 3-percentage-point margin of error for each candidate individually becomes approximately a 6-point margin of error for the difference between the two. This means that although we have observed a 5-point lead for the Republican, we could reasonably expect their true position relative to the Democrat to lie somewhere between –1 and +11 percentage points. The Republican would need to be ahead by 6 percentage points or more for us to be confident that the lead is not simply the result of sampling error.

In Poll B, which also has a 3-point margin of error for each individual candidate and a 6-point margin for the difference, the Republican lead of 8 percentage points is large enough that it is unlikely to be due to sampling error alone.

How do I know if there has been a change in the race?

With new polling numbers coming out daily, it is common to see media reports that describe a candidate’s lead as growing or shrinking from poll to poll. But how can we distinguish real change from statistical noise? As with the difference between two candidates, the margin of error for the difference between two polls may be larger than you think.

In the example in our graphic, the Republican candidate moves from a lead of 5 percentage points in Poll A to a lead of 8 points in Poll B, for a net change of +3 percentage points. But taking into account sampling variability, the margin of error for that 3-point shift is plus or minus 8 percentage points. In other words, the shift that we have observed is statistically consistent with anything from a 5-point decline to an 11-point increase in the Republican’s position relative to the Democrat. This is not to say such large shifts are likely to have actually occurred (or that no change has occurred), but rather that we cannot reliably distinguish real change from noise based on just these two surveys. The level of observed change from one poll to the next would need to be quite large in order for us to say with confidence that a change in the horse-race margin is due to more than sampling variability.

Even when we do see large swings in support from one poll to the next, one should exercise caution in accepting them at face value. From Jan. 1, 2012, through the election in November, Huffpost Pollster listed 590 national polls on the presidential contest between Barack Obama and Mitt Romney. Using the traditional 95% threshold, we would expect 5% (about 30) of those polls to produce estimates that differ from the true population value by more than the margin of error. Some of these might be quite far from the truth.

Yet often these outlier polls end up receiving a great deal of attention because they imply a big change in the state of the race and tell a dramatic story. When confronted with a particularly surprising or dramatic result, it’s always best to be patient and see if it is replicated in subsequent surveys. A result that is inconsistent with other polling is not necessarily wrong, but real changes in the state of a campaign should show up in other surveys as well.

The amount of precision that can be expected for comparisons between two polls will depend on the details of the specific polls being compared. In practice, almost any two polls on their own will prove insufficient for reliably measuring a change in the horse race. But a series of polls showing a gradual increase in a candidate’s lead can often be taken as evidence for a real trend, even if the difference between individual surveys is within the margin of error. As a general rule, looking at trends and patterns that emerge from a number of different polls can provide more confidence than looking at only one or two.

How does the margin of error apply to subgroups?

Generally, the reported margin of error for a poll applies to estimates that use the whole sample (e.g., all adults, all registered voters or all likely voters who were surveyed). But polls often report on subgroups, such as young people, white men or Hispanics. Because survey estimates on subgroups of the population have fewer cases, their margins of error are larger – in some cases much larger.

A simple random sample of 1,067 cases has a margin of error of plus or minus 3 percentage points for estimates of overall support for individual candidates. For a subgroup such as Hispanics, who make up about 15% of the U.S. adult population, the sample size would be about 160 cases if represented proportionately. This would mean a margin of error of plus or minus 8 percentage points for individual candidates and a margin of error of plus or minus 16 percentage points for the difference between two candidates. In practice, some demographic subgroups such as minorities and young people are less likely to respond to surveys and need to be “weighted up,” meaning that estimates for these groups often rely on even smaller sample sizes. Some polling organizations, including Pew Research Center, report margins of error for subgroups or make them available upon request.

What determines the amount of error in survey estimates?

Many poll watchers know that the margin of error for a survey is driven primarily by the sample size. But there are other factors that also affect the variability of estimates. For public opinion polls, a particularly important contributor is weighting . Without adjustment, polls tend to overrepresent people who are easier to reach and underrepresent those types of people who are harder to interview. In order to make their results more representative pollsters weight their data so that it matches the population – usually based on a number of demographic measures. Weighting is a crucial step for avoiding biased results, but it also has the effect of making the margin of error larger. Statisticians call this increase in variability the design effect .

It is important that pollsters take the design effect into account when they report the margin of error for a survey. If they do not, they are claiming more precision than their survey actually warrants. Members of the American Association for Public Opinion Research’s Transparency Initiative (including Pew Research Center) are required to disclose how their weighting was performed and whether or not the reported margin of error accounts for the design effect.

It is also important to bear in mind that the sampling variability described by the margin of error is only one of many possible sources of error that can affect survey estimates. Different survey firms use different procedures or question wording that can affect the results. Certain kinds of respondents may be less likely to be sampled or respond to some surveys (for instance, people without internet access cannot take online surveys ). Respondents might not be candid about controversial opinions when talking to an interviewer on the phone , or might answer in ways that present themselves in a favorable light (such as claiming to be registered to vote when they are not).

For election surveys in particular, estimates that look at “ likely voters ” rely on models and predictions about who will turn out to vote that may also introduce error. Unlike sampling error, which can be calculated, these other sorts of error are much more difficult to quantify and are rarely reported. But they are present nonetheless, and polling consumers should keep them in mind when interpreting survey results.

• Election 2016
• Methodological Research
• Nonprobability Surveys
• Survey Basics
• Survey Methods
• Telephone Surveys
• U.S. Elections & Voters

Andrew Mercer is a senior research methodologist at Pew Research Center .

Charting Congress on Social Media in the 2016 and 2020 Elections

Confronting 2016 and 2020 polling limitations, q&a: after misses in 2016 and 2020, does polling need to be fixed again what our survey experts say, a voter data resource: detailed demographic tables about verified voters in 2016, 2018, a majority of americans continue to favor replacing electoral college with a nationwide popular vote, most popular.

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 |  Media Inquiries

Research Topics

• Email Newsletters

ABOUT PEW RESEARCH CENTER  Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of  The Pew Charitable Trusts .

© 2024 Pew Research Center

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Thorac Dis
• v.9(10); 2017 Oct

Using the confidence interval confidently

Biomedical research is seldom done with entire populations but rather with samples drawn from a population. Although we work with samples, our goal is to describe and draw inferences regarding the underlying population. It is possible to use a sample statistic and estimates of error in the sample to get a fair idea of the population parameter, not as a single value, but as a range of values. This range is the confidence interval (CI) which is estimated on the basis of a desired confidence level. Calculation of the CI of a sample statistic takes the general form: CI = Point estimate ± Margin of error, where the margin of error is given by the product of a critical value (z) derived from the standard normal curve and the standard error of point estimate. Calculation of the standard error varies depending on whether the sample statistic of interest is a mean, proportion, odds ratio (OR), and so on. The factors affecting the width of the CI include the desired confidence level, the sample size and the variability in the sample. Although the 95% CI is most often used in biomedical research, a CI can be calculated for any level of confidence. A 99% CI will be wider than 95% CI for the same sample. Conflict between clinical importance and statistical significance is an important issue in biomedical research. Clinical importance is best inferred by looking at the effect size, that is how much is the actual change or difference. However, statistical significance in terms of P only suggests whether there is any difference in probability terms. Use of the CI supplements the P value by providing an estimate of actual clinical effect. Of late, clinical trials are being designed specifically as superiority, non-inferiority or equivalence studies. The conclusions from these alternative trial designs are based on CI values rather than the P value from intergroup comparison.

Introduction

Biomedical research is seldom done with entire populations but rather with samples drawn from a population. There are various strategies for sampling, but, wherever feasible, random sampling strategies are to be preferred since they ensure that every member of the population has an equal and fair chance of being selected for the study. Random sampling also allows methods based on probability theory to be applied to the data.

Although we work with samples, our goal is to describe and draw inferences regarding the underlying population. Values obtained from samples are referred to as ‘sample statistics’ which we have to use to garner idea of corresponding values in the underlying population, that are referred to as ‘population parameters’. But how do we do this? If we are doing ‘census’ type of studies, then the measured values are directly the population parameters since a census covers the entire population. However, if we are studying samples, then what we have in our hand at study end are the sample statistics. If we have a large enough and adequately representative sample, it is logical to presume that the sample statistics would be close to the ‘true values’, that is the population parameters, but they would probably not be identical to them. Strictly speaking, without doing a census it is not possible to get true population values. Practically speaking, it is possible to use a sample statistic and estimates of error in the sample to get a fair idea of the population parameter, not as a single value, but as a range of values. This range is the confidence interval (CI). How well the sample statistic estimates the underlying population value is always an issue. The CI addresses this issue because it provides a range of values which is likely to contain the population parameter of interest.

The CI is a descriptive statistics measure, but we can use it to draw inferences regarding the underlying population ( 1 ). In particular, they often offer a more dependable alternative to conclusions based on the P value ( 2 ). They also indicate the precision or reliability of our observations—the narrower the CI of a sample statistic, the more reliable is our estimation of the underlying population parameter. Wherever sampling is involved, we can calculate CI. Thus we can calculate CI of means, medians, proportions, odds ratios (ORs), relative risks, numbers needed to treat, and so on. The concept of the CI was introduced by Jerzy Neyman in a paper published in 1937 ( 3 ). It has now gained wide acceptance although many of us are not quite confident about it ( 4 ). To be fair it is not an intuitive concept but requires some reflection and effort to understand, calculate and interpret correctly. In this article we will look at these issues.

Meaning of CI

The CI of a statistic may be regarded as a range of values, calculated from sample observations, that is likely to contain the true population value with some degree of uncertainty.

Although the CI provides an estimate of the unknown population parameter, the interval computed from a particular sample does not necessarily include the true value of the parameter. Therefore, CIs are constructed at a confidence level, say 95%, selected by the user. This implies that were the estimation process to be repeated over and over with random samples from the same population, then 95% of the calculated intervals would be expected to contain the true value. Note that the stated confidence level is selected by the user and is not dependent on the characteristics of the sample. Although the 95% CI is by far the most commonly used, it is possible to calculate the CI at any given level of confidence, such as 90% or 99%. The two ends of the CI are called limits or bounds.

CIs can be one or two-sided. A two-sided CI brackets the population parameter from both below (lower bound) and above (upper bound). A one-sided CI provides a boundary for the population parameter either from above or below and thus furnishes either an upper or a lower limit to its magnitude.

Calculation of CIs

Formulas for calculating CIs take the general form:

The point estimate refers to the statistic calculated from sample data. The critical value or z value depends on the confidence level and is derived from the mathematics of the standard normal curve. For confidence levels of 90%, 95% and 99% the z value is 1.65, 1.96 and 2.58, respectively. The standard error depends on the sample size and the dispersion in the variable of interest.

Calculation of the CI of the mean is relatively simple. Here the formula is:

If we are calculating the 95% CI of the mean, the z value to be used would be 1.96. Table 1 provides a listing of z values for various confidence levels. The margin of error depends on the size and variability of the sample. Naturally, the error will be smaller if the sample size (n) is large or the variability of the data [standard deviation (SD)] is less and this is reflected in the SEM.

Ideally the SD used in the calculation should be the population SD. However, this is often unknown and if we are dealing with a reasonably large (say n >100, or at least >30) random sample, then the sample SD can be used as a fair approximation of the population SD. If the sample is small and one has to rely on the sample SD, then this requires derivation of the CI using the t distribution rather than the z value from the normal distribution. In this situation, the z value is to be replaced with the appropriate critical value of the t distribution with (n–1) degrees of freedom. Note that the Student’s t distribution, resembles the normal distribution, although its precise shape depends on the sample size. The required t value can be found from a t distribution table included in most statistical textbooks. For example, if the sample size is 25, the critical value for the t distribution that corresponds to a 95% confidence level with 24 degrees of freedom, is 2.064.

Let us now work out an example. The mean systolic blood pressure (SBP) and diastolic blood pressure (DBP) of 72 randomly selected chest physicians aged over 50 is 134 and 88 mmHg, with SD of 5.2 and 4.5 mmHg. What is the 95% CIs for the blood pressure readings?

Here we will consider the sample SD as fair approximation to the population SD. Therefore:

Thus the chest physicians appear to be non-hypertensive at present although they still need to keep a regular check on their blood pressure readings.

Unlike numerical variables, categorical variables are summarized as counts or proportions, and we will now deal with CI of a proportion. The formula for this is a bit more intimidating, but is still manageable for manual calculation.

Let us work through an example. A radiology resident has done a small observational study to find out the sensitivity and specificity of pleural effusion detected on digital chest X-rays (CXR) in predicting the risk of malignancy among subjects presenting with suggestive clinical findings. Her data is summarized in Table 2 .

The sensitivity of his diagnostic modality is [63/(63+25)]×100, i.e., 71.59%, while specificity is [53/(33+53)]×100, i.e., 61.63%; The 95% CI for the sensitivity would be:

And the 95% CI for the specificity would be:

In this same example, the odds of malignancy being present when pleural effusion is detected on CXR is 63/33, i.e., 1.9091, while the odds when pleural effusion is absent is 25/53, i.e., 0.4717. Therefore the OR for detecting malignancy when pleural effusion is present, compared to when it is absent, would be 1.9091/0.4717, i.e., 4.047.

Calculation of the 95% CI of the OR requires a more complicated formula, where we first derive the natural logarithm (log to base e, or ln) of the sample OR and then calculate its standard error. From this we derive the two confidence limits of the ln(OR), and then take their antilog to derive the 95% CI of the OR.

The formula is:

Where, SE(ln(OR)) = ( 1 a + 1 b + 1 c + 1 d )

Thus in the above example ln(OR) would be ln(4.047)  i . e .,  1.3980

The SE of ln(OR) would be ( 1 / 63 + 1 / 33 + 1 / 25 + 1 / 53 ) = ( 0.1050 )   i . e . ,   0.3241

The 95% confidence limits for ln(OR) would be 1.3980  ±  1.96  × 0.3241 = 1.3980  ±  0.6352  i . e .,  0.7628  t o  2.0332

Taking antilog of these two boundaries, the 95% CI of OR of 4.047 in this case would be 2.144 to 7.638. Note that since this 95% CI of the OR does not span the value 1, it implies that a pleural effusion detected by CXR in a patient with clinical examination findings suggestive of malignancy is approximately 2.1 to 7.6 times as likely to indicate malignancy than when it is not detected.

The reason why we resort to such an apparently complicated formula is that ORs are not normally distributed. They tend to be skewed towards the lower end of possible values. As a result, we must take the natural log of the OR and first compute the confidence limits on a logarithmic scale, and then convert them back to the normal OR scale. The same applies to relative risks and other sample statistics that are skewed in this manner.

Calculating CIs for distribution free statistics

From the examples above, it should be evident that CI take the general form of Point estimate ± Margin of error, where the margin of error is calculated by multiplying a critical value selected as per the required confidence limits with the standard error.

Standard errors cannot be calculated for distribution free statistics. Nevertheless, CIs can be calculated and have the same interpretation, that is they will present a range of values with which the true population value is compatible. In this case, the confidence limits are not necessarily symmetric around the sample estimate and are given by actual values in the sample that are chosen from the applicable formula.

The formula for calculating the CI of the median is:

For example, if there are 100 values in a sample data set, the median will lie between 50th and 51st values when arranged in ascending order. Applying the formula shown above, the lower 95% confidence limit is indicated by 40.2 rank ordered value, while the upper 95% confidence limit is indicated by 60.8 rank ordered value. Since there are no actual 40.2 and 60.8 ranked values, we choose the ranks nearest to these and values of these ranks then provide the approximate 95% CI for the median. For the 100 value series, this will therefore be the range indicated by the 40th to 61st rank ordered value.

For large samples, the CI for the median and other quartiles can be determined on the basis of the binomial distribution. You can see examples online ( 5 ).

Factors affecting the width of the CI

The width of the CI indicates the utility of our estimation of the population parameter. Suppose the weather forecaster uses the concept of the 99% CI to declare that tomorrows maximum temperature is going to be anywhere between 1 to 50 °C. It is extremely unlikely that he or she will be wrong but for you and me it would be utmost perplexing as to what to wear outdoors tomorrow. On the other hand if he says that the range is likely to be 20 °C to 30 °C on the basis of the 95% CI, then he has more chance of being wrong but it is easier for us to decide how we dress tomorrow. The factors affecting the width of the CI include the desired confidence level, the sample size and the variability in the sample.

The width of the CI varies directly with the confidence level. A 99% CI would be wider than the corresponding 95% CI from the same sample. This stands to reason, since a larger likelihood of containing the true population value would lie with the wider interval.

A larger sample size expectedly will lead to a better estimate of the population parameter and this is reflected in a narrower CI. The width of the CI is thus inversely related to the sample size. In fact, required sample size calculation for some statistical procedures is based on the acceptable width of the CI.

Variability in a random sample directly influences the width of the CI. A larger spread implies that it is more difficult to reliably estimate population value without large amounts of data. Thus as the variability in the data (often expressed as the SD) increases, the CI also widens.

Practical use of the CI

In descriptive statistics, CIs reported along with point estimates of the variables concerned, indicate the reliability of the estimates. The 95% confidence level is often used, though the 99% CI are used occasionally. At 99%, the width of the CI will be larger but it is more likely to contain the true population value, than the narrower 95% CI. Bioequivalence testing makes use of the 90% CI. In such studies, we can conclude that two formulations of the same drug are not different from one another if the 90% CI of the ratios for peak plasma concentration (C max ) and area under the plasma concentration time curve (AUC) of the two preparations (test vs. reference) lies in the range 80–125%.

Conflict between clinical importance and statistical significance is an important issue in biomedical research. Clinical importance is best inferred by looking at the effect size, that is how much is the actual change or difference. However, statistical significance in terms of P only suggests whether there is any difference in probability terms ( 6 ). One way to combine statistical significance and effect sizes is to report CIs. If a corresponding hypothesis test is performed, the confidence level is the complement of the level of significance, that is a 95% CI reflects a significance level of 0.05, while at the same time providing an estimate of the ‘true’ value. Indeed, if there is no overlap on comparing 95% CI surrounding point estimates of the outcome variable in different groups, once can conclude that statistically significant difference exists. On the other hand, even if there is small overlap, the difference between groups may not be clinically significant, irrespective of the P value. Thus stating the CI shifts the interpretation from a qualitative judgment about the role of chance to a quantitative estimation of the biologic measure of effect ( 7 ).

Of late, clinical trials are being designed specifically as superiority, non-inferiority or equivalence studies. The conclusions from these alternative trial designs are based on CI values rather than the P value from intergroup comparison ( 8 ). CI around the outcome point estimate for the test drug must fall wholly within a predefined equivalence margin on both sides of the line of no difference for establishing equivalence. For establishing non-inferiority, the lower bound of the 95% CI for the test drug must not cross the non-inferiority margin set a priori . For establishing superiority, the lower bound of the 95% CI for the test drug must lie beyond the line of no difference, while the upper bound extends beyond the superiority margin set a priori . Figure 1 summarizes these situations graphically. Selection of these margins has to be done with due care based on clinical judgment.

The positioning of 95% confidence limits around the point estimate in the test intervention group to establish non-inferiority, equivalence or superiority in clinical trials. The line of no difference is indicated by zero; negative delta indicates the non-inferiority margin or the lower bound of the equivalence margin, while positive delta indicates upper bound of the equivalence margin or the superiority margin.

The concept of CIs and confidence levels are also used in the calculation of sample size for prevalence surveys. The margin of error selected by the surveyor determines the acceptable deviation between the prevalence in the surveyed section of the population and the prevalence in the entire population. Thus, the margin of error implies a CI. The confidence level selected indicates how often the percentage of the population that has the condition of interest is likely to lie within the boundaries decided by the margin of error. Table 3 indicates how required sample size for population surveys varies with acceptable margin of error and confidence level.

Sample size is larger for a lower margin of error or higher level of confidence. Once the estimated population size is very large (>100,000), the sample size is not changing much.

Misconceptions regarding the CI

A 95% CI does not mean that 95% of the sample data lie within that interval. A CI is not a range of plausible values for the sample, rather it is an interval estimate of plausible values for the population parameter.

It is natural to interpret a 95% CI as a range of values with 95% probability of containing the population parameter. However, the proper interpretation is not that simple. The true value of the population parameter is fixed, while the width of the 95% CI based on a random sample will also vary randomly. If we take repeated random samples of equal size from the population, we will get a corresponding number of 95% CI values not all of which will contain the population parameter—in fact only 95% of them can be expected to contain the population parameter value. Thus the CI may not always give an idea of the population parameter.

Selection of the acceptable confidence level is arbitrary. We often use the 95% CI in biological sciences, but this is a matter of convention. A much higher level is often used in the physical sciences. For instance the six sigma concept, the quality improvement program that Motorola originated, and which is now popular in many manufacturing companies, utilizes a confidence level of 99.99966% ( 9 ). The engineers want to eliminate all risk of manufacturing poor-quality products and therefore work at this level of precision.

Finally, it is worthwhile to remember that the concept of the CI was introduced to provide an answer to the vexing issue in statistical inference of how to deal with the uncertainty inherent in results derived from data that represent randomly selected subset of a population. There are other answers, notably that provided by Bayesian inference in the form of credible intervals. Calculation of the conventional CI depends on set rules that ensure that the interval determined by the rule will include the true value of the population parameter. This is the so called ‘frequentist’ approach. The Bayesian approach offers intervals that can, subject to acceptance of interpretation of ‘probability’ as Bayesian probability, be interpreted as meaning that the specific interval calculated from a given dataset has a particular probability of including the true value, conditional on the particular situation ( 10 ). Bayesian intervals treat their bounds as fixed and the estimated parameter as a random variable, whereas conventional approach treats confidence limits as random variables and the population parameter as a fixed value. Unlike the Bayesian method, the frequentist method of computing CIs does not make use of any other prior information regarding the location of the population parameter. Thus, there is a philosophical difference between the two approaches which we can address at a later stage.

Acknowledgements

Conflicts of Interest: The author has no conflicts of interest to declare.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 21 May 2024

Pay researchers to spot errors in published papers

• Malte Elson 0

Malte Elson is an associate professor of the psychology of digitalization at the University of Bern, Switzerland.

You can also search for this author in PubMed   Google Scholar

You have full access to this article via your institution.

In 2023, Google awarded a total of US$10 million to researchers who found vulnerabilities in its products. Why? Because allowing errors to go undetected could be much costlier. Data breaches could lead to refund claims, reduced customer trust or legal liability.

It’s not just private technology companies that invest in such ‘bug bounty’ programmes. Between 2016 and 2021, the US Department of Defense awarded more than US$650,000 to people who found weaknesses in its networks .

Just as many industries devote hefty funding to incentivizing people to find and report bugs and glitches, so the science community should reward the detection and correction of errors in the scientific literature. In our industry, too, the costs of undetected errors are staggering.

Retractions are increasing, but not enough

That’s why I have joined with meta-scientist Ian Hussey at the University of Bern and psychologist Ruben Arslan at Leipzig University in Germany to pilot a bug-bounty programme for science, funded by the University of Bern. Our project, Estimating the Reliability and Robustness of Research (ERROR), pays specialists to check highly cited published papers, starting with the social and behavioural sciences (see go.nature.com/4bmlvkj ). Our reviewers are paid a base rate of up to 1,000 Swiss francs (around US$1,100) for each paper they check, and a bonus for any errors they find. The bigger the error, the greater the reward — up to a maximum of 2,500 francs.

Authors who let us scrutinize their papers are compensated, too: 250 francs to cover the work needed to prepare files or answer reviewer queries, and a bonus 250 francs if no errors (or only minor ones) are found in their work.

ERROR launched in February and will run for at least four years. So far, we have sent out almost 60 invitations, and 13 sets of authors have agreed to have their papers assessed. One review has been completed , revealing minor errors.

I hope that the project will demonstrate the value of systematic processes to detect errors in published research. I am convinced that such systems are needed, because current checks are insufficient.

Structure peer review to make it more robust

Unpaid peer reviewers are overburdened , and have little incentive to painstakingly examine survey responses, comb through lists of DNA sequences or cell lines, or go through computer code line by line. Mistakes frequently slip through. And researchers have little to gain personally from sifting through published papers looking for errors. There is no financial compensation for highlighting errors , and doing so can see people marked out as troublemakers.

Yet failing to keep abreast of this issue comes at a huge cost. Imagine a single PhD student building their work on an erroneous finding. In Switzerland, their cumulative salary alone will run to six figures. Flawed research that is translated into health care, policymaking or engineering can harm people. And there are opportunity costs — for every grant awarded to a project unknowingly building on errors, another project is not pursued.

Like technology companies, stakeholders in science must realize that making error detection and correction part of the scientific landscape is a sound investment.

Funders, for instance, have a vested interest in ensuring that the money that they distribute as grants is not wasted. Publishers stand to improve their reputations by ensuring that some of their resources are spent on quality management. And, by supporting these endeavours, scientific associations could help to foster a culture in which acknowledgement of errors is considered normal — or even commendable — and not a mark of shame.

How ‘research impact bonds’ could transform science funding

I know that ERROR is a bold experiment. Some researchers might have qualms. I’ve been asked whether reviewers might exaggerate the gravity of errors in pursuit of a large bug bounty, or attempt to smear a colleague they dislike. It’s possible, but hyperbole would be a gamble, given that all reviewer reports are published on our website and are not anonymized. And we guard against exaggeration. A ‘recommender’ from among ERROR’s staff and advisory board members — none of whom receive a bounty — acts as an intermediary, weighing up reviewer findings and author responses before deciding on the payout.

Another fair criticism is that ERROR’s paper selection will be biased. The ERROR team picks papers that are highly cited and checks them only if the authors agree to it. Authors who suspect their work might not withstand scrutiny could be less likely to opt in. But selecting papers at random would introduce a different bias, because we would be able to assess only those for which some minimal amount of data and code was freely available. And we’d spend precious resources checking some low-impact papers that only a few people build research on.

My goal is not to prove that a bug-bounty programme is the best mechanism for correcting errors, or that it is applicable to all science. Rather, I want to start a conversation about the need for dedicated investment in error detection and correction. There are alternatives to bug bounties — for instance, making error detection its own viable career path and hiring full-time scientific staff to check each institute’s papers. Of course, care would be needed to ensure that such schemes benefited researchers around the world equally.

Scholars can’t expect errors to go away by themselves. Science can be self-correcting — but only if we invest in making it so.

Nature 629 , 730 (2024)

doi: https://doi.org/10.1038/d41586-024-01465-y

Reprints and permissions

Competing Interests

The author declares no competing interests.

Related Articles

• Scientific community
• Research management
• Peer review

How I run a virtual lab group that’s collaborative, inclusive and productive

Career Column 31 MAY 24

Biomedical paper retractions have quadrupled in 20 years — why?

News 31 MAY 24

What is science? Tech heavyweights brawl over definition

Researcher parents are paying a high price for conference travel — here’s how to fix it

Career Column 27 MAY 24

How researchers in remote regions handle the isolation

Career Feature 24 MAY 24

Who will make AlphaFold3 open source? Scientists race to crack AI model

News 23 MAY 24

Plagiarism in peer-review reports could be the ‘tip of the iceberg’

Nature Index 01 MAY 24

Algorithm ranks peer reviewers by reputation — but critics warn of bias

Nature Index 25 APR 24

Associate Editor, High-energy physics

As an Associate Editor, you will independently handle all phases of the peer review process and help decide what will be published.

Homeworking

American Physical Society

Postdoctoral Fellowships: Immuno-Oncology

We currently have multiple postdoctoral fellowship positions available within our multidisciplinary research teams based In Hong Kong.

Hong Kong (HK)

Centre for Oncology and Immunology

Chief Editor

Job Title:  Chief Editor Organisation: Nature Ecology & Evolution Location: New York, Berlin or Heidelberg - Hybrid working Closing date: June 23rd...

New York City, New York (US)

Springer Nature Ltd

Global Talent Recruitment (Scientist Positions)

Global Talent Gathering for Innovation, Changping Laboratory Recruiting Overseas High-Level Talents.

Beijing, China

Changping Laboratory

Postdoctoral Associate - Amyloid Strain Differences in Alzheimer's Disease

Houston, Texas (US)

Baylor College of Medicine (BCM)

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies