Sampling Error- SE(p) Overview

Sampling Error

Research error is not an easy topic to discuss because it involves a variety of concepts, words, and formulas.  Because so many different types of people read this column, I have tried to simplify this discussion so just about anyone can understand it.

However, with that said, you may want to get a 6-pack of your favorite beverage because this answer is a bit long.  OK, so on to your questions . . .

All research that analyzes a sample selected from a population has some degree of sampling error or standard error, both of which are abbreviated "SE."  Sampling error is the estimated difference between the data from a sample and the "true" or "real" value that would be found if a census (analyzing the entire population) were conducted.  There is no sampling error when a census is conducted because the calculations are based on the entire population, but a census still contains measurement error and random error, so even a census is not a perfect measurement.

Now, before I send you to the Arbitron sampling error tables, I need to explain a few basics about sampling error.  First, there are two terms or concepts that important in sampling error:  Confidence Interval andConfidence Level.

Confidence Interval is the "±" percentage you hear about when a research study is conducted.  For example, a TV news anchor may report the results of a political poll and say something like, "The results of the poll have a margin of error of plus or minus 5 points (or 5 percent).  The plus or minus 5 percent is the Confidence Interval for the study.

Confidence Level is also a percentage and provides an indication of how confident a person can be with the results of a study.  For example, the 95% Confidence Level indicates that a person can be 95% certain that the results fall within the range of the Confidence Interval; the 99% Confidence Level indicates that a person can be 99% certain that a study's results fall within the range of the Confidence Interval.  Most behavioral researchers use the 95% confidence level; researchers in "hard" sciences of medicine, physics, and chemistry often use Confidence Intervals as high as 99.99% (or .0001).

Another way to understand the Confidence Level is that it provides an indication of how likely it is that the results of a study are due to error or chance.  For example, the 95% Confidence Level means that there is a 5% probability that the results of the study are due to error, chance, or both.  In other words, the higher the Confidence Level, the lower the probability that the results are spurious (false); the lower the Confidence Level, the more likely that the results are due to error or chance.  (Most behavioral science researchers use the 95% or 99% Confidence Level, although the 90% Level is occasionally be used in exploratory research—studies where the investigator is in the preliminary stages of an investigation. 

There is no rule related to the Confidence Level used in a research project.  Literally any Confidence Level can be used, and the decision to use one Confidence Level over another is purely arbitrary.  The most fundamental reason for selecting one Confidence Level is the desire for accuracy (remember . . . the higher the Confidence Level, the less likely the probability that results are due to error or chance).

For example, let's say that a radio researcher uses the 90% Confidence Level.  This means that the researcher (or client) is willing to accept the fact that there is a 10% probability that the results are due to error or chance.  There is nothing wrong with this as long as everyone involved in the research knows and understands the "rules" of the study.  However, researchers in the hard sciences would never use such a low Confidence Level.  As mentioned, most researchers in the hard sciences often use Confidence Levels of 99.99%, which means that there is only one chance in 1,000 that the results are due to error or chance.  Can you imagine the criticism a cancer researcher would receive if the results from a study testing for new drug were tested at the 90% Confidence Level?  Gag me.

I hope I haven't lost anyone yet, because this is important stuff if you want to read, interpret, and understand any research, including Arbitron and Nielsen ratings.

Summary up to this point . . . When you read any research study, you need to know the Confidence Interval and the Confidence Level.  The report, or the researcher, should say something like, "The results of this study have a Confidence Interval of ±5% at the 95% Confidence Level."  If you don't know these two pieces of information, you don't know how accurate the results are . . . ya gots ta know.

OK, Now we need to look at the formula for sampling error (remember, this is the Confidence Interval).  The formula for the Confidence Interval at the 95% Confidence Level:

where p is the result from the study, N = sample size, and Z represents the area under the normal curve related to the desired Confidence Level.

So, if 20% of the respondents from a sample of 500 give the same answer, the formula for the 95% confidence level (discussed below) is:

 

The 20% response has a maximum estimated sampling error of ±3.5%, which means that the "actual" or "real" percentage for the answer is somewhere between 16.5% and 23.5%.

Now, if you wanted to interpret the results at the 90% Confidence Level, you would multiply the result under the square root symbol times 1.64; at the 99% Confidence Level, multiply times 2.57.

Note:  The corresponding Z-scores in the formula come from a table known as the "Areas Under the Normal Curve."  If you want to see the table, I prepared one our textbook, Mass Media Research: An Introduction—just click here.  By the way, the Z-score for the 68% Confidence Level is 1.0, so there is no need to multiply the result under the square root.

So why is the discussion about Confidence Levels important?  Because the sampling error tables supplied by Arbitron are calculated at the 68% Confidence Level, which means that you can only be 68% certain that the data you interpret fall within the range of your Confidence Interval.  In other words, there is a 32% chance that the results you are reading are due to error or chance.

One more thing . . . The reason I explained the sampling error formula is that it is discussed on Arbitron's sampling error pages, but the formula is in prose, not numbers and symbols.  You need to know the origin of the sampling error.

So . . . Arbitron does have sampling error tables on the Internet, and you can view them by clicking here.

That's the answer to your first question.  On to the second . . .

You also ask what sampling error would I consider "OK," and what would cause me concern.  Well, we're dealing with behavioral research here, and humans are very difficult to describe and their behavior is particularly difficult to predict.  I don't like to exceed a Confidence Interval of 5%...

Extra credit opportunity

If you attend the event below, you will receive +3 points added to your first exam score: 

Active Minds at Ithaca College is hosting "Change the Conversation," a presentation and discussion event that examines the ways that the mass media portray issues of mental health and illness. We're interested in helping students become more aware and responsible consumers and creators of media content, and believe that the event applies particularly well to students in the TV-R department.

"Active Minds Presents... Change the Conversation" takes place on February 17 (Tuesday) from 7pm to 8pm in Park 279. Students can (but are not required to) indicate their plan to attend at https://www.facebook.com/events/1419041251722233/.

Review Sheet #1 MMRM

Review Sheet MMRM Exam #1

Spring 2015

Science and Research

Definition of research

Characteristics of scientific method (public, objective, empirical, systematic, cumulative, predictive)

Typical steps of the research process

Validity and Reliability

Importance of replication

Variables (intervening, confounding)

Theory

Hypotheses

Different ways of knowing

Elements of Research

Concept/construct

IV and DV

Four levels of measurement (nominal, ordinal, interval, ratio)

Major scales (Likert, semantic differential, simple)

Major types of validity (face, predictive, concurrent, construct)

Some major threats to validity

Hypotheses vs. Research Questions

Methodologies: Experiments, Surveys, Content Analysis

Sampling

Census vs. Sample

Probability vs. nonprobability

Random, systematic random, convenience samples

Sampling Error (Margin of Error)

How to compute Sampling Error

Confidence Level vs. Confidence Interval

What is the appropriate CL for most social science research (as a %)?

Appropriate z-score value

Ethics

Three major theoretical approaches (Deontological, Teleological, Relativistic)

Four major principles (autonomy, nonmaleficence, beneficence, justice)

Four special concerns of social science research: voluntary participation, informed consent, concealment, deception

Practice Problems

Here is a practice SE problem (margin of error):

Nielsen is interested in seeing if the Northeast has different viewing patterns than the rest of the country, so they systematically choose a random sample of 900 homes and measure the TV ratings. They note that the rating for Sunday Night Football is 21. What is the SE(p)? What is the confidence interval and what is the appropriate confidence level for social science? 

Here is a sample SD problem:

An executive at SyFy wants to know which new original movie to air this spring. Her choices are between two made-for-tv movies-- "SharkZombie" or "VampNado."

The first show was tested across the country and returned acceptance scores of

50

50

50

40

20

30 

(on a scale of 1-60), and the second show returned scores of 

50

40

40

40

30

40

Which show should she choose and why?

Here is a sample z-Score problem:

Which show had the better year, American Idol or X-Factor? How do you know?

Average rating of American Idol: 6.9

SD: 2.1

This year's rating of American Idol: 7.2

Average rating of The Voice: 7.5

SD: 2.0

This year's rating of The Voice: 7.8

Steps in the Development of a Research Project

Steps to the Scientific Method

 1. Select a problem.

 2. Review existing research and theory (when relevant).

 3. Develop hypotheses or research questions.

 4. Determine an appropriate methodology/research design.

 5. Collect relevant data.

 6. Analyze and interpret the results.

 7. Present the results in an appropriate form.

 8. Replicate the study (when necessary).

Sample Problems

Here is a practice SE problem (margin of error):

Nielsen is interested in seeing if the Northeast has different viewing patterns than the rest of the country, so they systematically choose a random sample of 800 homes and measure the TV ratings. They note that the rating for Sunday Night Football is 25. What is the SE?

Here is a sample SD problem:

An executive at ABC wants to know which sitcom to air this fall. Her choices are between two shows-- "Welcome home, son!" or "Married with Children... Again."  

The first show was tested across the country and returned acceptance scores of 5,5,5,4,1 (on a scale of 1-5), and the second show returned scores of 5,4,4,4,3. 

Which show should she choose and why?

Here is a sample z-Score problem:

Which show had the better year relative to expectations, The Bachelor or The Bachelorette? How do you know?

Average rating of The Bachelor: 6.1

SD: 4

This year's rating of The Bachelor: 7.1

Average rating of The Bachelorette: 4.5

SD: 2

This year's rating of The Bachelorette: 5.3

Sampling Overview

Some scientific research examines every member of a population, a process called a census.

However, in most situations, it is impractical to examine every member of a population. In these instances, researchers draw a sample from the population. A sample is defined as “a subset of the population that is representative of the entire population.”

There are two types of samples: probability samples and nonprobability samples.

A probability sample follows mathematical guidelines and allows researcher to calculate the estimated sampling error present in a study.

Probability samples include random samples, where each subjects in a population has an equal chance of being selected; systematic random samples, stratified samples, and cluster samples.

In addition, there are several types of nonprobability samples, including available samples, volunteer samples, purposive samples, and quota samples.

Sampling methods must be selected carefully by considering issues such as the purpose of the study, cost versus value, time constraints, and the amount of acceptable sampling error.

Think of probability (or representational) sampling along the following lines--

Suppose you were going to make a giant pot of chicken noodle soup-- 10 gallons. You throw in egg noodles, chicken, broth (chicken stock), potatoes, onions, celery, salt, pepper, carrots, etc. You cook the soup for hours and now it's time to get a sense of how it tastes.

If you were to consume the entire 10 gallons of soup before you declare how it tastes, we would call that a census. You could have, of course, just taken a ladle and tasted that. There is a catch, though-- the soup must be mixed up really well so that your ladle is a fair representation of the entire 10-gallon pot.

If you dip your ladle in and pull out a fair representation of noodles, chicken, broth, potatoes, onions, celery, salt, pepper, etc., then we conclude that the rest of the soup tastes the same.

Now let's assume that we accidentally scrape our ladle against all the built up gunk along the side of the pot before we pull it up. In this case, we taste our ladle full of soup and we are NOT impressed. It tastes disgusting and, in fairness, is not really a fair representation of what the soup tastes like. If, however, we don't take a different ladle full, we'll never know that. In this instance, our ladle is NOT a fair representation of the entire 10-gallon pot.

Well, this is what we do in science, too.

We don't need to measure all 310 million Americans to get a sense, for example, of what they watched on TV last night (trust me-- American Idol was the #1 show). Instead, as long as we take a ladle full of Americans (assuming that the pot-- in this case, the country-- is all mixed up), we should be able to get a taste of the entire population.

This is precisely what the Nielsen Media Research Company does when it reports the overnight TV ratings. Rather than asking ALL Americans what they watched, they ask only a small, representational sample and that's enough to get a sense as to what the rest of the country is doing, too.

How small? The TV ratings (representing what 110 million households watch) are generally garnered by asking a few thousand homes.

You may be thinking-- "There is NO WAY that a few thousand homes can adequately represent what 110 million households watched last night!"

Well, the data would suggest otherwise. It is quite accurate.

Let's look at the previous presidential election for further evidence.

A few days before the 2008 election, CNN surveyed voters across the country. Out of a country of 310 million people, they asked ONLY 714 potential voters who they planned on voting for in the presidential election.

Here were their reported results:

Sample size: 714

Obama-- 53%
McCain--46%

Meanwhile, in a DIFFERENT survey asking DIFFERENT people (but still representative), the McClatchy Group (a company that owns several newspapers) conducted its own poll and asked 760 likely voters who they planned on voting for and here were their reported results:

Sample size: 760

Obama--53%
McCain--46%

As for the REAL results from the election? Here they are--

Obama-- 52.9%
McCain--45.6%

How many people voted?

Just under 123 million...

Amazing, isn't it?

CNN was able to ask just over 700 people, yet they got the SAME results as they would have had they asked 123 MILLION people.

That, my friends, is the beauty of sampling.

It is science's best friend (social sciences and the natural sciences).

Is it fool-proof? Nope. That's why each legitimate survey sample will report a Margin of Error and a Confidence Level.

In social science, most confidence levels are 95%. Furthermore, we tend to use 3.5% as an acceptable margin of error.

In the CNN case, the margin of error is 3.5%. This means that CNN was 95% confident that the ACTUAL results would be +/- 3.5%. This is to say that they were confident that they actual results would be something like this--

Obama-- 49.5% - 56.5%

McCain-- 42.5% - 49.5%

As it turns out, their REAL margin of error was 0.1 for Obama and 0.4 for McCain.

Ok, now for our final project (which you'll work on alone, but I'll help with each step), we'll use a random sample of magazine covers (that I'll provide). We'll follow the same logic that CNN did. Rather than look at thousands of covers, we'll only look at dozens (but, if we do our job well, it will be representational of the overall population).

Get it?


Jack