7/20/2011

Costs error optimization for cell-landline dual frame surveys

Ana Slavec and Vasja Vehovar University of Ljubljana, Slovenia

Problem

 Growing segment of cell-only population  Cell phone surveys are difficult to conduct  Solution: Dual frames of landline and cell phones

Groves et al. (2004)

 What is the optimal design?

2

Research question

 x – mixture parameter (share of landline units in a dual frame survey)

n nF nM n x n (1 x)

1. How to analitically determine the mixture parameter in dual frame surveys? 2. What is its span in practice? 3. What is the span in various circumstances; which factors are the most influential?

3

1 7/20/2011

5 telephone use domains

Possession Availability n1 WF1xn Fixed-only

Fixed-only* n W xn Is available by 2 F 2 fixed phone

Has fixed phone Overlap* n3 WF 3 xn WM 3 (1 x)n Has mobile phone Is available by Mobile -only* mobile phone

n4 WM 2 (1 x)n Mobile-only

n5 WM 1 (1 x)n

4

Cost-error optimization

 Deming (1953), Kish (1965) and Groves (1989)

 Cost per unit of accuracy COST COST COST MSE accuracy 1 MSE  Mean square error (Hansen et al. 1953) 2 W S 2 MSE Var( p) Bias( p)2 h h (E( p ) P ) n h h h

 Costs (k=cost per unit, c=cost ratio) COST xnk (1 x)nkc kn (x (1 x)c)

5

Analytical solution

 Suppose all bias is due to incorrect allocation across sample strata and there is no nonresponse

5 W 2S 2 W 2S 2 W 2S 2 W 2S 2 W 2S 2 MSE( p) Var( p) Var( p ) 1 1 2 2 3 3 4 4 5 5 h i h n1 n2 n3 n4 n5

 Minimisation of product of variance and costs

W 2 S 2 W 2 S 2 W 2 S 2 W 2 S 2 W 2 S 2 f (x) 1 1 2 2 3 3 4 4 5 5 x c 1 x WF1 x WF 2 x WF3 xn wM 3 (1 x) WM1 (1 x) WM 2 (1 x)

x3 H x2 I xJ K f (x) x3 L x2 M xN

x 4 (HM IL) 2x3 HN JL x 2 IN JM 3KL 2xKM KN f '(x) 2 x3 L x 2 M xN

6

2 7/20/2011

7

Data (Flash Eurobarometer 2008)

Country Fixed only Overlap Mobile only Total AT 12,4 43,4 44,2 100 BE 13,0 64,5 22,5 100 DE 17,1 78,1 4,9 100 ES 21,1 56,7 22,2 100 FR 16,8 69,1 14,2 100 IT 7,9 53,7 38,4 100 PT 12,8 43,4 43,8 100 SI 12,3 71,3 16,3 100

• Sample size appr. 1000 per country;

• 60-70% units interviewed by fixed phone

8

Variables

 c1. Generally speaking, do you think that your country's membership in the European Union is a good thing or a bad thing?  c2. Generally speaking, do you think that having the euro is a good or bad thing for your country?  q12. Governments in all euro area countries are implementing various structural changes, often called reforms. Would you agree or disagree with the following statements related to such reforms?  a. There is a need for significant reforms to improve the performance of our economy.  b. I think successful reforms in other euro area countries put pressure on our government to reform.  c. Governments need to save more today in order to prepare public finances for the ageing of populations.  d. The government should increase taxes to finance economic reforms  e. The government should reduce expenditures, e.g. social benefits to finance economic reforms.  f. The EU should play an active role in the reform process in [country]  q16a. How has your household income changed since last year? Did it increase, decrease or stayed the same?  q16b. When looking into the future, how do you expect your household income change this year? Will it increase, decrease or stay the same?

9

3 7/20/2011

Data preparation

Fonly* WF 2 Monly* WM 2

Fonly* Overlap* Monly* Overlap Overlap* WF 3 Overlap* WM 3

10

Results

 Minimum of function f(x)=Var*Cost for different countries, variables (e.g. c1) and cost ratios (e.g. 1:3)

5

4

3 AT BE DE

2 ES f(x)=Var*Cost FR IT 1 PT SI

0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 x (share of fixed)

11

Optimal mixture for different cost ratios

1 1.2 1.6 3 6 10 AT 0.24 0.25 0.29 0.36 0.45 0.52 BE 0.44 0.47 0.51 0.59 0.67 0.73 DE 0.54 0.57 0.62 0.70 0.78 0.83 ES 0.48 0.51 0.55 0.63 0.71 0.77 FR 0.47 0.50 0.55 0.64 0.73 0.78 IT 0.24 0.26 0.30 0.38 0.48 0.55 PT 0.23 0.25 0.28 0.36 0.45 0.52 SI 0.42 0.45 0.50 0.61 0.71 0.77

12

4 7/20/2011

Influential factors

 No effect: Variable, data source.  Moderate effect: Cost ratio, country (population share).

1

0.8 DE FR ES BE SI 0.6

AT IT PT 0.4

Average Mixture parameter (x) parameter Mixture Average 0.2

0 0 0.5 1 1.5 2 2.5 3 3.5

M-only : F-only population share

13

Conclusions

 Gentle slope around the optimum value;  2 groups of countries:  SI, ES, BE, FR and DE: sample more fixed phone unit (if c=3: optimal mixture parameter 0.6 to 0.7)  PT, IT, and AT: sample more mobile phone units (if c=3: optimal mixture parameter about 0.35)  Key factor is the cost ratio.  If there is no difference in costs taking more fixed phone units is optimal for all countries (except DE);  If mobile phone surveys are much more expensive, having more fixed units is optimal also in PT, IT and AT.

14

Future research

 Uniform formulation of the questions about landline and cell phone use and its inclusion in more surveys;  Search for better population data to more accurately estimate strata weights;  A more accurate cost ratio estimate;  Elaboration of some problematic suppositions:  including bias in the MSE*Cost function;  considering response rates.  Comparison to three-stratum optimization.

15

5 7/20/2011

Literature

Biemer, Paul and Lars E Lyberg. 2003. Introduction to survey quality. Hoboken, NJ: Wiley. Blumberg Stephen J and Jullian V Luke. 2009. Wireless substitution: Early release of estimates from the National Health Interview Survey, July-December 2008. National Center for Health Statistics. Available from: http://www.cdc.gov/nchs/nhis.htm (accessed July 29 2010) Brick, J. Michael, Sarah Dipko, Presser Stanley, Clyde Tucker, and Yuan Yangyang. 2006. Nonresponse bias in a Dual Frame Sample of Cell and Landline Numbers. Public opinion Quarterly 70 (5, Special Issue). Available at: http://poq.oxfordjournals.org/cgi/content/ abstract/70/5/780 (accessed October 26 2009). Brick, J. Michael, Sherman W. Edwards, and Lee Sunghee. 2007a. Sampling telephone numbers and adults, interview length, and weighting in the California health interview survey cell phone pilot study. Public Opinion Quarterly, 71(5), 793-813. Brick 2009. 2009. Dual Frame Theory Applied to Landline and Cell Phone Surveys. Survey Research Methods Section Webinar, American Association for Public Opinion Research, (November 10). Brick, Michael J., Ismael F. Cervantes, Sunghee Lee and Greg Norman. 2010. Nonsampling Errors in Dual Frame Telephone Surveys. Unpublished paper. Callegaro, Mario, Charlote Steeh, Trent D. Buskirk, Vasja Vehovar, Vesa Kuusela, and Linda Piekarski. 2007. Fitting disposition codes to mobile phone surveys: experiences from studies in Finland, Slovenia and the USA. Journal of the Royal Statistical Society: Series A (Statistics in Society), 170(3), 647-670. Cochran, William G. 1978. Sampling techniques (3rd ed.). New York, NY: Wiley. Deming, Edwards W. 1950. Some theory of sampling. New York, NY: John Wiley & Sons. Deming, Edwards W. 1953. On a probability mechanism to attain an economic balance between the resultant error of response and the bias of nonresponse. Journal of the American Statistical Association 48(264), 743–772. Dolnicar, Sara, Christian Laesser, and Katrina Matus. 2009. Online Versus Paper: Format Effects in Tourism Surveys. Journal of Travel Research, 47(3), 295-316. European Commission. Public Opinion. Methodology – Instrument Description. Available at: http://ec.europa.eu/public_opinion/description_en.htm (July 29 2010). European Commission. 2008a. Special Eurobarometer 293. E-communication Household Survey. TNS Opinion and social. Available at: http://ec.europa.eu/public_opinion/archives/ebs/ebs_293_full_en.pdf (July 29 2010). European Commission. 2008b. Flash Eurobarometer 251. Public attitudes and perceptions in the Euro area. Gallup Europe, Brussels [Producer]. GESIS, Cologne [Publisher]. ZA4743, data set version 2.0.0. Description: http://info1.gesis.org/dbksearch13/sdesc2.asp?no=4743&db=e&doi=10.4232/1.10007 (July 29 2010). European Commission. 2009. Flash Eurobarometer 25. Public attitudes and perceptions in the Euro area. Analytical Report. Gallup Europe, Brussels. Available at: http://ec.europa.eu/public_opinion/flash/fl_251_en.pdf (July 29 2010). Eurobarometer. Available at: http://ec.europa.eu/public_opinion/standard_en.htm (January 15 2010). European Social Survey. Available at: http://www.europeansocialsurvey.org/ (January 15 2010). Gallup Europe. 2009: Alec M. Gallup Future of Survey Research Forum: Incorporating Mobile Phones in Social and Policy-oriented Surveys. Brussels, October 14. Groves, Robert M. (1989). Survey errors and survey costs. Hoboken, NJ: Wiley. Groves, Robert M., Floyd J. Fowler, Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau. 2004. Survey methodology. Hoboken, NJ: Wiley. Hansen, Morris H., William N. Hurwitz, and William G. Madow. 1953. Sample survey methods and theory. New York, NY: Wiley. Harter, Rachel M., Traci Mach, John D. Wolken, and Janella Chapline. 2007, June. Determining subsampling rates for nonrespondents. Paper presented at the Third International Conference on Establishment Surveys, Montréal, Canada. Hartley, Hermann Otto. 1962 Multiple Frame Surveys. Proceedings of the Social Statistics Section, American Statistical Association. Hideg, Gergerly, and Robert Manchin. 2005. The Inclusion of Mobile-only Persons in the Finnish ICS. Gallup Europe. Available at: http://www.europeansafetyobservatory.eu/doc/The%20Inclusion%20of%20Mobile-only%20Persons%20in%20the%20Finnish%20ICS.pdf (July 20 2010). Holt, D, and Smith, TMF. 1979. Post-stratification. Journal of the Royal Statistical Society 142 (1), pp.33-46.

16

Kuusela, Vesa, Mario Callegaro and Vasja Vehovar. 2008. The Influence of Mobile Telephones on Telephone Surveys. In: Advances in Survey Methodology. Wiley. Labour Force Survey. Available at: http://circa.europa.eu/irc/dsis/employment/info/data/eu_lfs/index.htm (January 15 2010). Little, Roderick A. J., and Donald B. Rubin. 1987: Statistical analysis with missing data. Lepkowski, James. M. and Robert M. Groves. 1984. The Impact of Bias on Dual Frame Survey Design. AMSE model - Proceedings of the Section on Survey Research, American Statistical Association. Available at: http://www.amstat.org/sections/SRMS/ Proceedings/papers/1984_051.pdf (accessed October 26 2009). Lepkowski, James. M. and Robert M. Groves. 1986. A Mean Squarred Error Model for Dual Frame, Mixed Mode Survey Design. Journal of the American Statistical Association 396 (81). Available at: http://www.jstor.org/stable/2289062 (accessed October 26 2009). Lessler, Judith T. and William D. Kalsbeek. 1992. Nonsampling error in surveys. New York: Wiley. Linacre, Susan J. and Dennis J. Trewin. 1993. Total survey design – Application to a collection of the construction industry. Journal of Official Statistics, 9(3), 611–621. Link, Michael W., Michael Battaglia, Martin P. Frankel, Larry Osborn, and Ali H. Mokdad 2007. Reaching the U.S. cell phone generation: Comparison of cell phone survey results with an ongoing landline telephone survey. Public Opinion Quarterly, 71(5), 814-839. Lyberg, Lars E., Lull Japec, and Paul Biemer. 1998. Quality improvement in surveys - A process perspective. Paper presented at the Survey Research Methods Section. Lynn, Peter and Dave Elliot. (2000). The British Crime Survey: A review of methodology. London, UK: National Centre for Social Research. Keeter, Scott. 2007. What’s Missing From National Landline RDD Surveys? The Impact of Growing Cell-Only Population. Public Opinion Quarterly 71 (5). Available at: http://poq.oxfordjournals.org/ cgi/content/abstract/71/5/772 (accessed October 26 2009). Kennedy, Courtney. 2007. Evaluating the Effects of Screening for Telephone Service in Dual Frame RDD Surveys. Public opinion Quarterly 71 (5). Available at: http://poq.oxfordjournals.org/cgi/content/abstract/71/5/750 (accessed October 26 2009). Kish, Leslie. 1965. Survey sampling. New York, NY: Wiley. Moschner, Meinchard. 2008. Flash Eurobarometer: Sampling and Fieldwork. GESIS (December 11). Available at: http://www.gesis.org/en/services/data/survey-data/eurobarometer-data- service/flash-eb/sampling-fieldwork/ (July 29 2010). Moschner, Meinchard. 2010. Flash Eurobarometer: Weighting Overview. GESIS (June 21). Available at: http://www.gesis.org/en/services/data/survey-data/eurobarometer-data-service/flash- eb/weighting-overview/ (July 29 2010). Vehovar, Vasja and Katja L. Manfreda. 1998. How many mailings are enough? In A. Koch & R. Porst (Eds.), Nonresponse in survey research (pp. 139-149). Mannheim: Zentrum für Umfragen, Methoden und Analysen. Vehovar, Vasja, Katja L. Manfreda, and Zenel Batagelj. 2001. Sensitivity of electronic commerce measurement to the survey instrument. International Journal of Electronic Commerce, 6(1), 31-52. Vehovar, Vasja, Eva Belak, Zenel Batagelj and Sanja Čikić. 2004. Mobile phone surveys: The Slovenian case study. Metodološki zvezki (Advances in Methodology and Statistics), 1(1), 1- 19. Vehovar Vasja, Nejc Berzelak in Katja L. Manfreda. 2010. Mobile Phones in an Environment of Competing Survey Modes: Applying Metric for Evaluation of Costs and Errors. Social Sciences Computer Review 28 (303). Weisberg, Herbert. F. 2005. The total survey error approach: A guide to the new science of survey research. Chicago: University of Chicago Press. Westling, Sara. 2008. A simulation approach to evaluate the cost efficiency of nonresponse follow-ups. Working Paper series, Handelshögskolan vid Örebro universitet, 2008(11). Retrieved from http://www.oru.se/oru-upload/Institutioner/Ekonomi%20statistik%20och%20informatik/Forskning/wp2008-11.pdf

17

Thanks for your attention! [email protected]

18

6 7/20/2011

Sample size

Budget n k x c(1 x)

Var *Cost g(x) n

19

Regression analysis

20

7