AJPH RESEARCH

Twitter-Based Detection of Illegal Online Sale of Prescription Opioid

Tim K. Mackey, PhD, MAS, Janani Kalyanam, PhD, MS, Takeo Katsuki, PhD, and Gert Lanckriet, PhD

Objectives. To deploy a methodology accurately identifying tweets marketing the inadequately addressed despite the passage illegal online sale of controlled substances. of the RHA. Specifically, tech- Methods. We first collected tweets from the Twitter public application program in- nology is now ubiquitous (e.g., 84% of terface stream filtered for prescription opioid keywords. We then used unsupervised American adults use the Internet and 65% machine learning (specifically, topic modeling) to identify topics associated with illegal use a social networking site), fueling the growth of dubious Internet Web online marketing and sales. Finally, we conducted Web forensic analyses to characterize sites, now estimated in the tens of thou- different types of online vendors. We analyzed 619 937 tweets containing the keywords sands globally.8 In a recent report, the codeine, Percocet, fentanyl, Vicodin, Oxycontin, oxycodone, and hydrocodone over National Association of Boards of Phar- a 5-month period from June to November 2015. macy reviewed more than 11 000 Web Results. < fi A total of 1778 tweets ( 1%) were identi ed as marketing the sale of sites and found that 96% were not in “ ” controlled substances online; 90% had imbedded hyperlinks, but only 46 were live at compliance with state and federal laws or the time of the evaluation. Seven distinct URLs linked to Web sites marketing or illegally the association’s patient safety and phar- selling controlled substances online. macy practice standards (e.g., they did not Conclusions. Our methodology can identify illegal online sale of prescription opioids require valid prescriptions or issued pre- from large volumes of tweets. Our results indicate that controlled substances are scriptions via online consultations or trafficked online via different strategies and vendors. questionnaires only), including 13% that Public Health Implications. Our methodology can be used to identify illegal online sellers dispensed controlled substances. in criminal violation of the Ryan Haight Online Pharmacy Consumer Protection Act. (Am J Previous published studies (including Public Health. 2017;107:1910–1915. doi:10.2105/AJPH.2017.303994) investigative reports by the US Govern- ment Accountability Office showing that Oxycontin [oxycodone], Percocet See also Bachhuber and Merchant, p. 1858. [oxycodone/paracetamol], and Vicodin were successfully purchased from online n February 2001, an 18-year-old honors reexamine and revise opioid-related policies – without a prescription) have Istudent from California died after pur- and guidance.2 5 confirmed the public health and patient chasing Vicodin (hydrocodone/acetamin- Policies aimed at curbing prescription safety dangers of illegal online sale of pre- ophen), a commonly abused prescription opioid abuse have focused on establishing – scription opioids.9 13 Also, recent studies opioid drug, from a “no prescription” guidelines to prevent inappropriate pre- have established an association between online pharmacy.1 His name was Ryan scribing, developing abuse deterrents, social media technologies and “no- Haight, and his untimely death led to pas- “ ” regulating pill mills, and preventing prescription” online pharmacy drug pro- sage of the 2008 Ryan Haight Online – drug diversion (such as through state motion and access.14 20 Hence, we sought Pharmacy Consumer Protection Act (RHA; monitoring pro- to build on prior research by employing HR 6353), a federal law that amended the 1,4,6,7 grams). However,theroleofthe an innovative methodology involving Controlled Substances Act (21 USC 801). Internet and its continued promotion of “big data,” machine learning, and Web The RHA was specifically designed to re- prescription opioid abuse remains forensic analyses to identify and characterize spond to the growing use of the Internet to illegally market and sell controlled sub- ABOUT THE AUTHORS stances directly to consumers.1 More than Tim K. Mackey is with the Department of Anesthesiology and Department of Medicine, University of California, San Diego, 15 years later, nonmedical use of pre- and the Global Health Policy Institute, San Diego. Janani Kalyanam is with the Global Health Policy Institute and the scription medications is a national epidemic, Department of Electrical and Computer Engineering, University of California, San Diego. Takeo Katsuki is with the Kavli Institute for Brain and Mind, University of California, San Diego. Gert Lanckriet is with the Department of Electrical and with the US Centers for Disease Control and Computer Engineering, University of California, San Diego. Prevention reporting that deaths attribut- Correspondence should be sent to Timothy Ken Mackey, PhD, MAS, Global Health Policy Institute, 8950 Villa La Jolla Drive, able to prescription opioids have more than Suite A203, San Diego, CA 92037 (e-mail: [email protected]). Reprints can be ordered at http://www.ajph.org by clicking the 2 “Reprints” link. quadrupled since 1999. This situation has This article was accepted July 7, 2017. prompted federal and state agencies to both doi: 10.2105/AJPH.2017.303994

1910 Research Peer Reviewed Mackey et al. AJPH December 2017, Vol 107, No. 12 AJPH RESEARCH

social media use by online pharmacies in a reasonable time frame (relative to manual tweets actively promoting the online mar- their efforts to promote the illegal sale of annotation by human coders).21,22 Specifi- keting and sale of prescription opioids directly prescription opioid drugs. cally, unsupervised methods such as topic to consumers (note that Twitter is not spe- models have proven to be useful in obtaining cifically an e-commerce platform but can a summary of the underlying themes pre- include tweets that promote and link to sent in large text corpora. We used a model external Web pages offering products for METHODS called the Biterm Topic Model (BTM), direct sale). We used a 3-step process (involving, as which is designed to detect themes and First, we reviewed each hyperlink to noted, cloud-based computing, machine patterns in the corpora of short texts (such determine whether it was still “live” (i.e., learning, and Web content analyses) to as tweets). We had previously used this whether there was a valid URL linked to identify and characterize illicit online vendors model to examine prescription drug abuse an active Web page). We then discarded 14,15 marketing the online sale of controlled sub- behavior and trends. tweets with “dead” hyperlinks (i.e., URLs stances via the popular microblogging site The BTM first detected a preconfigured linked to inactive Web pages or broken Twitter (which currently has 328 million number of themes from the filtered data hyperlinks). Second, we assessed the type of active users as of the first quarter of 2017). set of tweets containing the study pre- Web site associated with the hyperlink to With respect to data collection, we used scription opioid keywords. This process determine whether it engaged in marketing the cloud-based Amazon Web Services to produced a set of themes (or word group- of prescription opioid products or direct sale collect tweets through the Twitter public ings) that were then coded, via human an- streaming application program interface notation, to manually identify those clearly of prescription opioid drugs to consumers. (API). associated with prescription opioid mar- For Web sites categorized as marketing keting, distribution, or sale. For example, prescription opioid drugs, we then examined Web page hyperlinks to determine whether Data Collection themes with a combination of words in- cluding “[prescription opioid drug name],” any of them redirected to a site selling In our first step, we used Amazon Web “buy,”“cheap,”“price,” and “discount” prescription opioid drugs. Services EC2 t2.micro virtual instances (all identifiedasbeingusedbyonline For all Web sites categorized as selling preconfigured with RStudio to collect large pharmacies) were identified and extracted prescription opioid drugs (via either a hyper- volumes of tweets filtered for prescription for further analysis.8,14,23 Themes containing link directly embedded in a tweet or opioid keywords from the Twitter public words that were irrelevant (e.g., mentions a hyperlink in an associated Web page), streaming API, as detailed in a separate of news reports, individual substance abuse we then used 2 external databases to de- published study.15 We used this strategy behavior, or drug safety warnings from termine the site’s legal classification and to maximize data collection and generate the Food and Drug Administration) were domain registration information. In assessing a corpus of tweets more generalizable to discarded along with their associated fi the full Twitter “firehose.”15 Our keywords legal classi cations, we used the Internet tweets and were not included in sub- consisted of a combination of international security monitoring company LegitScript sequent analyses. Tim K. Mackey and nonproprietary names and brand names of LLC, which categorizes Internet pharmacy Janani Kalyanam coded the themes and “ ” commonly abused prescription opioids Web sites as rogue (vendors engaging in achieved high intercoder reliabilities for (earlier studies had shown that these key- illegal, unsafe, or misleading activity), “un- the BTM word grouping inclusion criteria words are commonly used by online phar- approved” (vendors with regulatory com- (k=0.98). macies for promotion purposes).14,15 Our pliance risks, such as those operating legally Importantly, this methodology allowed final list of keywords comprised codeine, in one jurisdiction but not in others), us to use an unassisted machine learning fentanyl, hydrocodone, oxycodone, Oxy- “unverified” (vendors not subject to Legit- algorithm to filter out hundreds of thou- contin, Percocet, and Vicodin. Tweets Script review or monitoring), or “legitimate” sands of tweets unrelated to the study’s ob- were collected over a 6-month period (vendors adhering to LegitScript criteria).15,24 jective and isolate tweets that specifically from June to November 2015. In determining site domain registration in- mentioned the marketing and sale of pre- scription opioid drugs (i.e., “signal” data). formation, we used a WHOIS lookup ’ Data Coding We then reviewed and characterized these Web tool to identify a Web site s Internet The second step involved the use of tweets in our forensic examination. Protocol (IP) address location and registry a machine learning protocol to isolate information. word groupings associated with tweets that Characteristics such as legal status and mentioned marketing and purported sale Forensic Examination Web site domain name registration are im- of prescription opioid drugs. In coding and In the third step, we performed content portant to identify given that, per US fed- characterizing large volumes of Twitter data analyses by analyzing tweets from signal eral law and the RHA, the online sale or (in the hundreds of thousands), the applica- data. We focused our forensic Web analyses foreign importation of controlled substances tion of machine learning is a critical strategy on tweets that included hyperlinks to exter- is illegal and could constitute a federal criminal to achieve scale and complete analyses in nal Web pages given our aim of identifying offense.

December 2017, Vol 107, No. 12 AJPH Mackey et al. Peer Reviewed Research 1911 AJPH RESEARCH

RESULTS and (4) 1 online classified advertisement “pain relievers” product category advertised We collected and analyzed 619 937 involving sale of controlled substances. the direct sale of a host of controlled sub- tweets containing our selected prescription Twitter accounts tweeting this content stances such as Percocet, Oxycontin, opioid drug keywords. Using the BTM consisted of 3 primary categories: rogue hydrocodone, Codogesic (codeine), and machine learning protocol, we identified online pharmacy Twitter accounts, user Vicodin. and annotated 1778 (0.003%) tweets as accounts used to retweet online pharmacy Another identified rogue online phar- “signal” data containing content associated advertisements, and user accounts that macy (located in Pakistan, with a US IP) with illicit online drug sales (the percentage retweeted but had no noticeable link to marketed itself as the “World Most Trusted of relevant tweets ranged from 0% to 95%; an online pharmacy. Health Care Online Mall” (example B in Table 1). These results indicate that the ab- A visual summary of our forensic ana- Figure A). The hyperlinks sent prospective solute volume of tweets related to illegal lyses is provided in the Figures A and B consumers to its Web page offering the sale promotion and sale of prescription opioid (available as a supplement to the online of codeine phosphate (a narcotic analgesic), drugs is relatively small in comparison with version of this article at http://www.ajph. and the site also advertised the sale of other Twitter communications related to other org), and a depiction of geographical IP prescription painkillers. The third online news, education, and behavioral aspects of addresses and Web site registrant addresses pharmacy characterized itself as a “Canadian prescription opioid drug abuse. For example, is provided in an interactive online Drugstore” (example C in Figure A), but in our prior analysis of a larger prescription Google map (see https://drive.google. its WHOIS information indicated an IP ad- opioid data set, we detected several million com/open?id=13qnaNCAKlVJVOez dress in the and a registrant tweets related to behavioral and risk trends TUObv3Szto1Q&usp=sharing). address in Pakistan. Although this site was via the BTM, including high volumes of categorized as rogue, we were unable to polydrug abuse discussions.14 A visual sum- detect sale of controlled substances; however, mary of our methods and main findings is Illicit Online Pharmacies other prescription drugs were sold without provided in Figure 1. The first category involved tweets con- the need for a valid prescription. In our examination of the signal data, taining live hyperlinks to illicit online phar- A second category included a link to an we detected 1608 tweets with embedded macies classified as rogue (Figure A). One online pharmacy (located in Pakistan, with hyperlinks; however, only 46 of these hy- of the most active marketers was a rogue a German IP) that was not listed in the perlinks (consisting of 7 distinct URLs) online pharmacy (example A in Figure A) LegitScript database (example D in Figure A) were still “live” at the time of our June marketing its services as an “online pharmacy but offered the sale of several therapeutic 2016 content analysis. An in-depth analysis shop” and “overseas online pharmacy” classes of prescription drugs (including of the 7 distinct URLs uncovered the (with a Canadian IP location and a regis- sleeping pills, antibiotics, injectable steroids, presence of (1) 3 “rogue” online pharmacies; trant address in Italy). The site exhibited and growth hormones), a live customer (2) 1 online pharmacy that could not be the typical characteristics of an illegal online service chat, discreet packaging, and sale verified through LegitScript but directly pharmacy, including the sale of prescription without a prescription. The site included “ sold controlled substances without a pre- drugs from multiple categories, overseas de- more than 30 different products in its an- ” scription; (3) 2 online pharmacies using livery (with international money transfers algesics/pain relievers category, including blogs, social media, user forums, and affili- directed to Pakistan), and sale without Percocet. The site also advertised the sale ate marketing to sell prescription opioids; a valid prescription.8,25 Importantly, the site’s of other dangerous and powerful controlled substances such as morphine, Valium (di- azepam), and ketamine. TABLE 1—Summary of Twitter Analysis: Online Pharmacies Promoting Illegal Sale of Prescription Opioid Drugs, 2015 Blogs, Social Media, User Forums, Total and Affiliate Marketing Tweets, Most Correlated Relevant Tweets With Links (Provided Live Distinct Drug Name No. Tweets, No. Tweets, % Tweet Is Relevant), % Links, % Links, No. A third category included illicit online pharmacies that used Internet marketing Codeine 431 625 874 48 68 3.1 2 strategies (Figure B). One rogue online Percocet 75 215 410 69 99 0.3 1 pharmacy (located in Pakistan, with a Cana- Fentanyla 28711 ...... dian IP) used the popular blogging platform Vicodin 28 610 286 95 100 1.4 2 blogspot.com (example E in Figure B) to post advertisements for Valium, Percocet, Oxycontin 27 734 465 59 99 2.9 2 and other controlled substances without Oxycodone 18 061 336 82 99 2.6 1 a prescription. The blog post included Hydrocodone 9 981 282 93 87 10.0 3 a hyperlink to the online pharmacy site “ ” aWe were not able to detect tweets correlated with illicit online pharmacy marketing through our Biterm and was advertised as a safe website. It Topic Model approach to content coding data. also used fake quality seals purporting that

1912 Research Peer Reviewed Mackey et al. AJPH December 2017, Vol 107, No. 12 AJPH RESEARCH

Step 1: Data Collection Step 2: Data Coding AWS, R, and Twitter Public Streaming API JSON Data Files: Objective: To identify Data collected via AWS instances Objective: To use machine learning tweets associated with 619 937 tweets containing to detect and extract tweets using streamR filltered for keywords collected from illegal online sale of keywords: codeine, fentanyl, associated with online marketing and controlled substances hydrocodone, oxycodone, June-November 2015 sale of controlled substances Oxycontin, Percocet, and Vicodin

Biterm Topic Model (BTM) Unsupervised Machine Learning Identification of word groupings: for theme detection: “[prescription opioid drug name]”, “buy”, “cheap”, “price”, 4 illicit online pharmacy Step 3: Forensic Examination “discount” (all adjectives and “selling hyperlinks a aruguments” used by online pharmacies) ( see online appendix) (Examples A, B, C, Da)

1 social media/blog account Objective: Perform a Signal Tweets: Human Annotation for Inclusion (Example E ) content analysis to 46 (7 distinct) “live” identify type of Web site, Relevant tweets (”signal data”) extracted and 1 user forum/affiliate legal status, and hyperlinks associated with coded for relevance including: (1) related to marketing network domin registration online sale of controlled online pharmacy; (2) contains hyperlink; and a information (3) conrtains “live” hyperlink. (Example F ) substances

1 online classifeds ad (Example Ea)

Note. AWS = Amazon Web Services; API = application programming interface; JSON = JavaScript Object Notation. aSee Figures A and B, available as supplements to the online version of this article at http://www.ajph.org, for examples A–F.

FIGURE 1—Summary of Methodology and Main Findings: Study of Online Pharmacies Promoting Illegal Sale of Prescription Opioid Drugs, 2015 the site was a “best choice,” a common located in Russia and the Netherlands that each tweet represents a potential patient marketing tactic for illicit online pharma- anonymized registration information. safety hazard and substance abuse risk given cies.8,25 In addition, the blog post embedded that all of the Web sites associated with the ’ “ ” a link to the site s Google+ social media Other Online Sources tweets were categorized as rogue or were clearly involved in “no-prescription” page and allowed users to tweet or favorite In addition, our Twitter surveillance de- sale of controlled substances, a direct violation (via Facebook) the post promoting access tected an individual user advertising the to controlled substances. “street” sale of controlled substances via an of the RHA. Thus, it appears that Twitter A second tweet linked to a thread on online classifieds Web site in the Hills District represents a viable modality for criminal a user forum associated with Indian music of Sydney, Australia (example G). The post actors to engage in the illegal marketing and advertising the sale of hydrocodone advertised the sale of numerous controlled and sale of prescription controlled sub- (example F in Figure B). The thread’s ad- substances with no prescription required, stances online, despite laws in the United fi vertisement redirected users to an affiliate express delivery, discreet packaging, and States and other countries speci cally pro- 1,26,27 marketing Web page (i.e., an online advertiser even door-to-door delivery services. Also, hibiting these practices. Our study’s methodology can also be that collects fees for redirecting user traffic the site offered discounts for bulk purchases adapted for other approaches aimed at to e-commerce sites) that listed 10 additional and directed prospective customers to detecting small volumes of Twitter discus- hyperlinks, all advertising the sale of hydro- submit an order via e-mail. sions that may be associated with criminal codone or other controlled substances health-related activities. This form of “ ” without prescription. Hyperlinks redirected “anomaly detection” when mining large users to 9 Web sites (7 rogue online pharmacies DISCUSSION volumes of unstructured social media data is and 2 online pharmacies selling controlled Our results indicate that a small percent- made possible through advances in machine substances without a prescription [but not age (< 1%) of tweets collected for a set of learning such as the BTM. The targeted listed in the LegitScript database]) and an common prescription opioid analgesic data generated from our “big data” strategy additional affiliate marketer. Registrant product keywords were associated with could also serve as an important digital tool addresses included Latvia, India, and online marketing or direct-to-consumer for stakeholders (such as law enforcement the United States, with some registrant sale of controlled substances. Even though personnel, drug regulators, and substance addresses protected by Web privacy services these tweets numbered only in the thousands, abuse researchers) actively engaged in the

December 2017, Vol 107, No. 12 AJPH Mackey et al. Peer Reviewed Research 1913 AJPH RESEARCH

fight against illegal online distribution and sale coding data for illicit online pharmacy to the current national prescription drug of controlled substances. themes and hyperlinks until approximately abuse epidemic. The methodology also Collectively, the illegal online pharmacies 8 months after we completed Twitter data has the potential to be used for surveillance assessed here used a variety of digital mar- collection. This was primarily attributable and detection of other health-related illegal keting strategies to promote their question- to our efforts to first examine prescription online activities, including the promotion able products and services. The catalyst for drug abuse behavior as detailed in our pre- and online sale of illicit and synthetic drugs.35 online promotion began with Twitter and viously published study using the BTM.14 As evidenced by our findings, extremely then expanded to the use of blogs, other This delay probably resulted in a higher questionable vendors are advertising highly social media platforms, user forums, fake percentage of signal data with “dead” addictive controlled substances and selling “quality” seals, and even local online classified hyperlinks, suggesting that our methodology them directly to consumers via Twitter advertisements. These results indicate that is most effective when used with real-time without the necessary oversight of a clini- online pharmacies may engage in multi- data as opposed to retrospective data. cian, regulatory agency, or state public channel digital marketing campaigns to reach Also, we did not examine non–English- health or law enforcement agency. At the consumers, including leveraging social media, language tweets (75 619 non-English tweets, time of the completion of our study, the as these communication forms are subject accounting for approximately 11% of our “rogue” Web sites we identified remained to lower standards of regulation and oversight entire prefiltered data set, were collected active online, continuing to place the than are Web search engines (e.g., in 2011, but discarded in our preanalysis data public at risk. Google settled with the US Department of cleaning process). In addition, hyperlinks Crucially, the unregulated online sale Justice for $500 million in connection with that redirected to live Web sites were of controlled substances to US consumers knowingly advertising no-prescription online reviewed for content at a specific point in is a direct violation of federal law under pharmacies that sold Oxycontin and Ritalin time after tweets had been collected. Hence, the RHA. Hence, our methodology can [methylphenidate] and has since taken cor- it is possible that the content provided on be useful in surveillance efforts, with rective action).8,28 the online pharmacies we reviewed changed violating Web sites reported directly to the In addition, we discovered through our between the time of data collection and DEA and the Food and Drug Administra- Web forensic analyses that the majority of the content analysis process. We addressed tion for additional enforcement action sites had non-US registrant addresses, with this limitation by reviewing active Web (including through the Food and Drug many linked to Pakistan. This is alarming page links (at the time of our content Administration’s BeSafeRx Web site, the given news reports that have identified analysis) and links to pages cached in DEA’s reporting form for suspicious online Pakistan as both a source and an exporter Google search engine results. We did not pharmacies, and the requirement that of fake, counterfeit, and falsified medications, detect any noticeable differences when both US-based Internet service providers including via online sales.29 Recently there live and cached pages were available remove illegal content or issue cease and have been at least 2 separate criminal con- and compared. desist letters). victions of Pakistani nationals arrested for Also, registrant addresses collected during Our methodology can also help ensure distribution of illicit controlled substances forensic analyses are of dubious validity. In that the RHA is better implemented, to Internet consumers.30,31 One of these some cases, the registrant address fields did not monitored, and enforced in a constantly indictments involved a Pakistani national match valid geographic locations or involved evolving digital environment in which who allegedly used Internet Web site ads mismatching data (e.g., a listed address in social media use is becoming ubiquitous and filled orders for other Internet pharmacy Pennsylvania but a listed zip code in Arizona). in a broad range of Internet-using pop- sites to illegally import, distribute, and sell Finally, because of the illegal nature of the ulations, including young people suscepti- prescription drugs sourced from Pakistan, transactions identified, we could not purchase ble to substance abuse.1 Improved RHA India, and China (including unapproved controlled substances and test them for au- monitoring and enforcement could better versions of the controlled substances phen- thenticity and potency. The practice of protect the Ryan Haights of the present termine, diazepam, alprazolam, and buying prescription drugs for a fictional pa- and future from the dangers of “digital” lorazepam).30 Hence, in addition to the risk tient and making payments to an illicit online substance abuse and addiction to prescription of dependence, abuse, overdose, and possible pharmacy raises serious ethical and legal painkillers. death, our analysis indicates that these challenges and is generally considered illegal CONTRIBUTORS Pakistan-affiliated sites may present a separate by the US Drug Enforcement Agency (DEA). T. K. Mackey and J. Kalyanam conducted the data ana- safety risk of exposing the public to fake lyses and wrote the initial draft of the article. T. K. Mackey – versions of medications.32 34 and T. Katsuki collected the data for the study. T. K. Public Health Implications Mackey, J. Kalyanam, and G. Lanckriet designed the study. All of the authors contributed to the formulation, We have described an innovative meth- drafting, and completion of the article. Limitations odology for filtering large volumes of There are limitations regarding the gen- Twitter content to specifically isolate tweets ACKNOWLEDGMENTS Tim K. Mackey and Takeo Katsuki received funding eralizability of our study results. For example, promoting the illegal online sale of con- for the data collection phase of this project from the Al- we did not begin the process of content trolled substances, an activity contributing liance for Safe Online Pharmacies (ASOP), a social

1914 Research Peer Reviewed Mackey et al. AJPH December 2017, Vol 107, No. 12 AJPH RESEARCH

welfare organization addressing the issue of illicit online 16. Hanson CL, Cannon B, Burton S, Giraud-Carrier 32. Mackey TK, Liang BA. The global counterfeit drug pharmacies. We greatly acknowledge this support. C. An exploration of social circles and prescription drug trade: patient safety and public health risks. J Pharm Sci. Note. Tim K. Mackey is a noncompensated member abuse through Twitter. J Med Internet Res. 2013;15(9):e189. 2011;100(11):4571–4579. of the ASOP academic advisory panel, and (as noted) 17. Hanson CL, Burton SH, Giraud-Carrier C, West JH, 33. Attaran A, Barry D, Basheer S, et al. How to achieve both Mackey and Takeo Katsuki received funding for Barnes MD, Hansen B. Tweaking and tweeting: explo- international action on falsified and substandard medi- this study from ASOP through a pilot research grant cines. BMJ. 2012;345:e7381. exploring online prescription drug abuse risks. The funder ring Twitter for nonmedical use of a psychostimulant drug had no role or input in the study. (Adderall) among college students. J Med Internet Res. 34. Braden JB, Edlund MJ, Sullivan MD. Suicide deaths 2013;15(4):e62. with opioid poisoning in the United States: 1999–2014. Am J Public Health. 2017;107(3):421–426. HUMAN PARTICIPANT PROTECTION 18. Liang BA, Mackey T. Direct-to-consumer advertising fi — No protocol approval was needed for this study because with interactive Internet media: global regulation and 35. To ghi B, Lee JD. Internet highs seizures after publicly available data were used. public health issues. JAMA. 2011;305(8):824–825. consumption of synthetic cannabinoids purchased online. J Addict Med. 2012;6(3):240–241. 19. Mackey TK. Global reach of direct-to-consumer REFERENCES advertising using social media for illicit online drug sales. 1. Mackey TK, Liang BA, Strathdee SA. Digital social J Med Internet Res. 2013;15(5):e105. media, youth, and nonmedical use of prescription drugs: 20. Liang BA, Mackey TK. Prevalence and global health the need for reform. J Med Internet Res. 2013;15(7):e143. implications of social media in direct-to-consumer drug 2. Centers for Disease Control and Prevention. Un- advertising. J Med Internet Res. 2011;13(3):e64. derstanding the epidemic. Available at: https://www.cdc. gov/drugoverdose/epidemic/index.html. Accessed 21. Yoon S, Elhadad N, Bakken S. A practical approach August 6, 2017. for content mining of tweets. Am J Prev Med. 2013;45(1): 122–129. 3. Volkow ND, McLellan AT. Opioid abuse in chronic pain—misconceptions and mitigation strategies. N Engl J 22. Cole-Lewis H, Varghese A, Sanders A, Schwarz M, Med. 2016;374(13):1253–1263. Pugatch J, Augustson E. Assessing electronic cigarette- relatedtweetsforsentimentandcontentusingsuper- 4. Frieden TR, Houry D. Reducing the risks of relief— the CDC opioid-prescribing guideline. N Engl J Med. vised machine learning. J Med Internet Res. 2015; 2016;374(16):1501–1504. 17(8):e208. “ 5. Meldrum ML. The ongoing opioid prescription epi- 23. Orizio G, Rubinelli S, Schulz PJ, et al. Save 30% if ” demic: historical context. Am J Public Health. 2016;106(8): you buy today : online pharmacies and the enhancement 1365–1366. of peripheral thinking in consumers. Pharmacoepidemiol Drug Saf. 2010;19(9):970–976. 6. Kattan JA, Tuazon E, Paone D, et al. Public health detailing—a successful strategy to promote judicious 24. Mackey TK, Aung P, Liang BA. Illicit Internet opioid analgesic prescribing. Am J Public Health. 2016; availability of drugs subject to recall and patient safety 106(8):1430–1438. consequences. Int J Clin Pharm. 2015;37(6):1076–1085. 7. Kennedy-Hendricks A, Richey M, McGinty EE, 25. Orizio G, Merla A, Schulz PJ, Gelatti U. Quality of Stuart EA, Barry CL, Webster DW. Opioid overdose online pharmacies and Websites selling prescription deaths and Florida’s crackdown on pill mills. Am J Public drugs: a systematic review. J Med Internet Res. 2011; – Health. 2016;106(2):291 297. 13(3):e74. 8. Mackey TK, Nayyar G. Digital danger: a review of the 26. Liang BA, Mackey T. Searching for safety: addressing global public health, patient safety and cybersecurity search engine, Website, and provider accountability for threats posed by illicit online pharmacies. Br Med Bull. illicit online drug sales. Am J Law Med. 2009;35(1): 2016;118(1):110–126. 125–184. 9. Bachhuber MA, Cunningham CO. Availability of 27. World Health Organization. Safety and security on buprenorphine on the Internet for purchase without a pre- the Internet: challenges and advances in member states. scription. Drug Alcohol Depend. 2012;130(1–3):238–240. Available at: http://www.who.int/goe/publications/ 10. Boyer EW, Wines JD. Impact of Internet pharmacy goe_security_web.pdf. Accessed August 6, 2017. regulation on opioid analgesic availability. J Stud Alcohol Drugs. 2008;69(5):703–708. 28. US Department of Justice. Google forfeits $500 million generated by online ads and prescription drug sales 11. Forman RF. Availability of opioids on the Internet. by Canadian online pharmacies. Available at: http:// JAMA. 2003;290(7):889. www.justice.gov/opa/pr/2011/August/11-dag-1078. 12. Forman RF, Woody GE, McLellan T, Lynch KG. html. Accessed August 6, 2017. The availability of Web sites offering to sell opioid 29. Somra G. The backstreet labs feeding Pakistan’s fake medications without prescriptions. Am J Psychiatry. 2006; 163(7):1233–1238. drug trade. Available at: http://www.cnn.com/2015/08/ 30/asia/pakistan-fake-drugs/index.html. Accessed Au- fi 13. US Government Accountability Of ce. Internet gust 6, 2017. pharmacies: federal agencies and states face challenges combating rogue sites. Available at: http://www.gao. 30. US Immigration and Customs Enforcement. Pakistani gov/assets/660/655751.pdf. Accessed August 6, 2017. man indicted and arrested for selling and distributing new, misbranded, and counterfeit prescription drugs. Available 14. Kalyanam J, Katsuki T, Lanckriet G, Mackey TK. Exploring trends of nonmedical use of prescription drugs at: https://www.ice.gov/news/releases/pakistani-man- and polydrug abuse in the Twittersphere using un- indicted-and-arrested-selling-and-distributing-new- supervised machine learning. Addict Behav. 2017;65: misbranded-and. Accessed August 6, 2017. 289–295. 31. US Department of Justice. Pakistani national sen- 15. Katsuki T, Mackey TK, Cuomo R. Establishing a link tenced in international counterfeit drug conspiracy. between prescription drug abuse and illicit online phar- Available at: https://www.justice.gov/usao-edtx/pr/ macies: analysis of Twitter data. J Med Internet Res. 2015; pakistani-national-sentenced-international-counterfeit- 17(12):e280. drug-conspiracy. Accessed August 6, 2017.

December 2017, Vol 107, No. 12 AJPH Mackey et al. Peer Reviewed Research 1915