Do Not Trust the Polls

Worse Than the Margin of Error

If you are one of the people who, despite the not insubstantial vig (cut of winnings + fee on withdrawal) play the PredictIt Markets, my first piece of advice is to be careful trusting the polls.

Many players think they can beat the PredictIt binaries with some rudimentary knowledge if the underlying races in question, and that is often true that you can get in on the positive side of a binary, but it really depends on the research that you are doing.

It’s also important to remember that winning and finding value are two different things. Just because a candidate has a high likelihood of winning doesn’t mean that you have great value on a bet.

That’s just some general gambling advice, but I’m not here to talk about what I think are good picks on PredictIt, or anywhere else, today. What I am talking about today is that the polls for state offices can often miss wildly, so if you are only counting on one (or a handful) of polls in making your bets, you might be in for a surprise!

Here is an article about a shocker in the Senate from 2014 as reported by 538:.

Another article from 2014 that suggests that the polls were skewed in favor of Democrats:.

In general terms, this may or may not be, but I would never bet against Nate Silver.

When it comes to specific elections and/or specific polls, there can be much different factors at play than what has been mentioned above. I reached out to fivethirtyeight once upon a time, but they didn’t respond, so I assume they had no interest on my insight into the matter. As a result, you get to read it here!

My Background

While it would hardly qualify as extensive work experience, I did work in a telemarketing call center for about two years, combined, over both times that I worked there. During that time, I worked as a telephone sales representative, as a trainer and as a supervisor. I also filled in on recruiting, for the training manager, quality assurance and even shift manager on a couple of occasions.

The Way Phone Polls Work

The fact is that phone polls, by their very nature, can often be flawed. It’s not because they don’t always talk to the right subset of respondents in a way most likely to be balanced and result in an accurate sampling of the population at large. It’s because, sometimes, the phone polls don’t even attempt to do that. We’re going to break this down into a few categories:

1. Intentional Polling Bias
2. Inadvertent Cheating
3. Intentional Cheating & Contracts
4. Cheating the System

1.) Intentional Polling Bias

The thing that readers must understand is that every client who asks for phone polling has different criteria for the respondents and tries to skew them in different ways. Ir has been widely reported.

That the order of names on the ballot can make a difference with some voters such that the name listed first is at a slight advantage. Polls operate the same way. Once again, this is something that the pollsters themselves are doing deliberately.

For one thing, polls are often contracted directly by the campaign, or the campaign contracts a research firm who then contracts the call center.

The name that appears first in the survey is only one way that the polls can be skewed in favor of one candidate or another. There are a few other things that the polls can do, as well:

In some instances, you will launch into a little pro-candidate speech that consists of about two or three paragraphs, and then ask whether or not the party to whom you are speaking is voting for that candidate. If they give any answer except a, “Yes,” you will then rebut by saying something else favorable about the candidate you want them to promise the vote to. If they again do not answer in the affirmative, you would then ask if they are voting for (other person). In other words, you would take the, “Yes,” answer for one candidate immediately, but then you attempt to defend that candidate over the other if the answer is not yes.

Some polls do the opposite of that, in which the first paragraph or two will trash a different candidate and then ask the person whether they will vote for the one for whom you are calling. If they say they are voting for the one you are trashing, then you try to rebut them by making a positive statement about your candidate, and then asking them again.

These calls are actually really funny when you are talking to someone who is actually following the race. Usually they will ask some specific question about where the candidate stands on an issue, and because the campaign doesn’t want you screwing up unless you have something scripted, you’re supposed to kindly thank the person for his/her time and terminate the call.

There will occasionally be polls during which people are asked about multiple races. Almost Universally, a certain party will be listed first when you are asking the questions. Again, you obviously have to list one or the other first in a given call, but a non biased polling mechanism would be to have a 50/50 split of who gets listed first.

I think that most people are going to expect the candidates or political action committees to construct their polling in such a way to make a favorable result more likely, so most of you haven’t learned anything new here. Just wait, because things are going to get good.

Polls that are landline only also tend to be slightly biased in favor of Republicans, who trend older, but if those are the phone numbers available based on voting data, what can you do?

2.) Inadvertent Cheating

Phone-based polling is always going to have something of a human factor to it, and the result can often lead to some mistakes with these types of calls.

It should be known that the call center where I was employed also sold credit cards and credit card fee products. We would be granted a certain number of calling hours for each, “Program,” and so our first priority was to hit that at as high an SPH (Sales Per Hour) as possible.

Needless to say, when we would get these political hours, the result would usually be open overtime. In other words, everyone was free to be on the phones as many hours that particular week as they wanted, because we had to maximize our profits by (hopefully) using every available calling hour.

The calling hours for following months with our credit card client, however, were largely going to be based on that SPH. You can probably guess that we didn’t exactly put our best and brightest telephone sales representatives on political. In many cases, being on political for the week would be something of a moratorium on the termination that was surely coming for low production, or other issues.

Beyond that, when we were inundated with available calling hours, sometimes we would, “Open Enroll,” people to do political calling. That’s right, it was a veritable, “Bring a friend to work day,” literally. These people would be slam-trained on the computer system, taught how to read the script and off and running after a supervisor confirmed they more-or-less had a rudimentary grasp on the spoken English language. They didn’t always have that. When you’re desperate, you’re desperate.

Anyway, the supervisors would, “Monitor,” the calls and would watch the screen of the reps in the meantime. You obviously can’t listen in on all of the calls at once, so you would get to people as best you could. Political was often given lowest priority anyway, as I said, not our, “Best and brightest.”

On one occasion, nobody had got to a person until an hour into the political campaign. I just so happened to be the one to monitor her and I noticed that she would not bring her cursor down to the correct response when the contact answered her. She went an entire call just clicking on whatever the first item listed was. When confronted, she said that she thought the verbal response was enough and she didn’t need to move the cursor to the appropriate response. I have no idea why she would think that because political calls were generally NOT recorded.

Anyway, I couldn’t really cut her loose from the campaign, despite having my doubts about the veracity of her claimed ignorance. Sometimes, you just need a body in a seat. Of course, it’s hard to tell how many people she entered as voting for Candidate A who said they were voting for Candidate B...and this was a multi-campaign political program!

The other thing is that we need to slam these political calls through as quickly as possible, because those often get paid just on how many completed polls have been done. Complete the polls, get paid, move on to the next (if applicable). The faster the polls get completed, the less we are paying the reps to do it because they are not there as long. (The average political campaign would last between 4-6 hours)

The result of needing to get done quickly is that we don’t have time for nuance! If we call a guy and his response is, “Yeah, I’ll vote for that lying, piece of crap, disgusting, filthy, cheating, thief and son of a dog Michaelson!!!” Good, he’s voting Michaelson. Thank you for your time, sir, and have a good night. He’d be halfway through saying, “I’m not actually voting for Micha-” when you’d hang up.

The thing is, we just want an answer and we don’t care what the answer is. The faster we can complete calls, the better. A very high percentage of our regular reps and political people were smokers, so we’d roll out, “Complete fifteen polls and get a free ten minute break!” I wonder if that ever made anyone complete a poll incorrectly?

The thing that you have to understand about telemarketing is that it is shady dealings, at least outbound telemarketing. It’s about to get a whole lot shadier.

3.) Intentional Cheating

First of all, we never intentionally cheated because we favored any one candidate, political party, et cetera, et cetera. We really didn’t care about any of that or what our respondents’ answers even were, except incoherent screaming about politics always drew a laugh!

As I mentioned before, all of these political programs were contract-based in one way or another. We would be given a set of, “Leads,” and that’s what we had to work with. What the telemarketing company was compensated was usually variable in one way or another, so let’s look at the different ways we were compensated:

A.) Total, “Contacts,” disposed. In some cases, we were just supposed to complete as many surveys as possible, regardless of the number. We’d get paid on the basis of how many were actually completed, so the incentive was to complete as many as possible until the file had been beaten so badly that the law of diminishing returns would take over and we were paying the reps more than they were generating per hour. You usually tried to stop just short of that happening.

B.) Percentage of, “Contacts,” disposed. In this one, they would give us a calling file and we would get paid incrementally higher amounts based upon the total percentage of contacts in that file that we had disposed. We would get paid for each one, but the amount that would be paid would be greater the greater the percentage we did.

C.) Strict Stopping Point. This was the most common one where a pollster might want something like 300 respondents, and then we were to stop immediately. It wouldn’t matter if we did more than 300, we wouldn’t get paid for any over that.

D.) Calling Hours. These were obviously the least intense and required no cheating whatsoever. This would be like our credit card client who would pay $XX.xx per calling hour and wanted us to call a certain number of hours, regardless of the result. If the program was really slowing down, sometimes they’d pull the plug, but the company always got paid on the number of hours the reps were on the phones.

Okay, so now it’s time for me to give you a crash course in how calling files work.

The way the calling files work is that we have what is called an auto-dialer, which if you’ll recall the old days, is why sometimes you would answer the phone and there would be a second or two of silence after you said, “Hello,” before the representative began his/her diatribe.

Anyway, there are only so many numbers in a calling file, but sometimes there are multiple, “Files,” per program. We usually tried to divide it up that way. When we would run out of, “Leads,” in the first program, we would, “Roll the reps into,” the second program and tell the first program to, “Recycle,” the numbers.

In the world of credit card calling, this tendency would piss people off, but not quite as badly because we couldn’t call the same lead within a four hour window unless our auto-dialer picked up an answering machine and hung up. Mandated by the Federal Communications Commission, you could also only have a certain percentage of, “No answers,” which is where a lead answers the phone, but there is no representative to greet them. The result is that we would have to have the auto-dialer set to keep the hangup percentage conservative. We also couldn’t call on Sundays, per the credit card client, or before or after certain times that were more strict than allowed by the FCC.

All of these things went out the window with political. We would call you as many times as we could and that hangup rate would be cranked up to eleven! We weren’t hanging up on 50% of the calls, or anything crazy like that, but it was usually between 2-3%. I know that doesn’t seem like much, but that’s actually crazy-high and would be against FCC Regs on the credit card program.

The result is that people were getting called several times, per hour, if the files were beaten up badly enough. Like I said, before, on some of these we have no choice because if we didn’t hit a certain number of disposed contacts, we wouldn’t get paid at all as a call center.

Think about the dynamics of this for a second: We get a call file with 1,000 phone numbers in it of which we have to dispose 300 primary contacts, on an early Saturday afternoon. We can’t always talk to any registered voter who answers the phone, (sometimes you can, depends what the client wants) so we need 30% of people just to be home on a Saturday afternoon to even have a prayer of doing this legitimately.

Not only do they have to be home, they have to be willing to do the poll. As I’ve stated before, many of these polls had a lead-in that would either praise one candidate or bash another (though some were neutral). In the meantime, if the lead asked anything that would put us off a scripted answer, we were supposed to politely terminate the call. Of course, we weren’t going to put in a disposition of, “Declined/Terminated,” because that would kick the number out and we wouldn’t be able to call it again. “Callback,” disposition it is! Talk to you in ten minutes, Joe from Kokomo!

We all know what’s coming, we need completed polls, and it starts to look hopeless at a certain point. Even if we do get to the desired number of completions, we’re going to lose money because of how many calling hours it’s been. Time for a few of our mantras:

Fake It To Make It!!!

If You Ain’t Lyin’, You Ain’t Tryin’!!!

We would allow an answering machine to complete a poll if the auto answering machine hangup didn’t catch it. (It didn’t catch most of them) We would allow a busy signal to complete a poll. We would allow someone who was not the lead to complete a poll. We would allow a child to complete a poll. We would allow the babysitter to complete a poll.

It didn’t matter because these polling companies to contract us didn’t understand how the game works, so don’t hate the player. If they’re not going to pay you because you didn’t hit the desired number of contacts, but that’s because their lead base (which they often wouldn’t tell you until the company accepted the contract) was garbage, there’s nothing you can do. We’re not going to not get paid!

Somebody screams about how many times you’ve called and slams the phone down? Easy: Complete the survey, dispose the completion. At least they won’t get called again. DO NOT CALL LIST!? That’s adorable. It’s political calling. DOES. NOT. APPLY. Talk to you in ten minutes, Katie from Schenectady!

The best was when we would get people who would ask, “If I do your stupid poll, will you not call me anymore?” Answer: Yes. That is the ONLY way we will stop calling you.

Anyway, we would usually take a look at who was paying for the polling and then make a decision on how we wanted to balance it. Either that, or we would tell the reps to put through the illegitimate completions in a balanced way or a way as close as they can remember to what was actually happening.

Sometimes more of our completions were illegitimate than legitimate, so who knows what the hell the voters were thinking? Sometimes we would look at other polls of the area that had been conducted and just tell the reps to do that, “Six for the one guy, four for the other.”

The only polls I can recall in which there wasn’t at least some cheating were one of these three scenarios, or multiple simultaneously:

• They actually gave us enough leads for the number of completions they wanted to see. Usually on political, you’d need about ten leads for every legitimate completion you wanted. That was true even if half of the actual people you talked to received the call positively. There are just a ton of people who aren’t home all the time.
• They were paying by the calling hour regardless of the results. This was the way to get the MOST accurate results. We would actually instruct the reps to take their time (obviously) and make sure to complete everything with the utmost accuracy. Which was fine, because that’s usually what that sort of client REALLY wanted, or they wouldn’t be paying that way.
• An individual or his/her spouse could take the survey. This effectively doubles the amount of leads. The best ones on this are when you don’t even have to ask for the contact, you can just ask whoever answers, “Are you the spouse of Johnny from Punxsutawney?”

4.) Cheating the System

Obviously, no client wanted us to outright cheat. I mean, maybe they did if you were cheating in favor of their party or candidate, but they’re not going to flat out tell you that.

The result is that we would occasionally have, “Client sessions,” so that the client could also listen in on the calls (with a supervisor, of course) and make sure everything was going correctly and the calls were being disposed properly.

In a perfect world, at least for the client, the supervisor would just pick a handful of random reps (usually three or five) and the client would wait and listen until each rep spoke to five or ten actual contacts.

Because we were not stupid supervisors, these reps were never random.

When I said our, “Best and brightest,” didn’t call political, that wasn’t quite true. Our best sellers and best on the phones would do client sessions for our credit card client (when we could get away with using the same reps) AND political. We’re not going to risk putting somebody on the phone who might be mumbling semi-coherently when we have skilled reps who can transition seamlessly from credit card sales into reading a political script verbatim and enthusiastically.

It also worked because, for them, it was a break. In fact, we’d usually toss them a free smoke break if they’d do really well.

How it would work is we would give the shift manager a list of who we needed, “Rolled into,” political and at what time. When the client had listened to enough calls, we’d roll that person back to the credit card program and switch to the next. Supervisors would always make it sound like they were picking randomly, “Okay, so that was Chris and he did fine...let’s see how this Emily is doing, I’ll tap us into her now.”

Sometimes it would be a contracted research company who subcontracted us and we would actually get an explicit go ahead from them to, “Do whatever you have to do to get the completes goal.” In other words, they’d sometimes tell us to cheat.

Even the Autocalls

I honestly even suspect the autocalls of cheating sometimes by being results-biased. The first thing is phrasing the questions in a biased way and the second thing is that I think they multi-call the houses of results they like.

For example, I just got a call last week with an automated survey that I completed. I have a soft spot for the dying industry, it put bread on the table for a couple total years, after all. Anyway, I got a call again the following day and five question in, I realized it was the exact same survey I had completed the previous day.

The reason they would do that is not because they think the same person is going to do the same survey multiple times, but because people who live together often have somewhat similar political leanings. It’s more likely than not. So, they’re basically trying to skew the responses by having both people in the same household do the survey since they think both parties will lean the same way.

Unfortunately, the results of these surveys are suspect for another reason: Absolutely anybody can do them. It gives you prompts to ask if you’re a registered voter, likely voter, etc. etc., but you can be a ten year old and hit the button if you want to.

Conclusion

Anyway, that’s why I would not trust or base my betting in state races off of an individual poll or even a subset of a few polls. In some cases, you end up with only a sample size of 1,500 voters across multiple polls, and some of those are probably the same people multiple times in different polls!

There are two types of people in the world:

Those who like to do surveys.

Those who do not like to do surveys.

There have been instances in which we thought two different research firms were using the same set of contacts, just based on there being a very similar number of leads, if not exact, and the call dispositioning percentages being eerily similar.

Beyond that, all of the cheating and the general shady nature of the telemarketing business, in general.

When I see a poll that’s an extreme outlier, I just laugh.

Then I remember, that might be the one poll that conducted itself the right way.