Data sources

We performed an extensive literature search in the online databases PubMed and Web of Science to identify publications that have used the TCS score(s) as an independent or dependent variable from 27 March 2006, when the first TCS was published, until 1 December 2019. The search was conducted using ‘tobacco control scale’ as the keyword in all fields without any other restrictions to ensure a very sensitive search. The Ethics and Clinical Research Committee of the Hospital Universitari de Bellvitge approved this study (PR247/18).

Study selection

We identified 69 publications (32 duplicated in both databases). After removing duplicates, two researchers (AF and AB) screened the titles and abstracts, obtaining 32 studies. The inclusion criteria were quantitative research and inclusion of the TCS score(s) (as dependent or independent variable) in the analysis. We found 27 eligible publications (Figure 1). We completed our search by manually reviewing the reference lists of the selected papers and by conducting the same search in Google Scholar (www.scholar.google.com; with search terms in English). These additional searches provided five new publications that met the inclusion criteria and the full-texts were reviewed.

Figure 1

Flowchart of the selection process of publications for full-text review

Therefore, we finally included 32 publications that used the TCS score(s) as a dependent or independent variable.

Data extraction

A detailed protocol and Microsoft Access^® database were designed to extract and register the information from each publication. The evaluation protocol was developed by three researchers who are experts in tobacco control (AF, CM, and EF). The protocol describing the main objectives, information sources, search strategy and eligibility criteria, and the data collection was reviewed and approved by all researchers. All variables for which data were described in the publications’ Methods sections were listed.

Two reviewers independently extracted the data according to the protocol (AF and AB). If any discrepancies emerged, the reviewers discussed the papers until agreement was reached and, when no consensus was met, divergences were solved by discussing them with a third reviewer (CM). The evaluation process was conducted in January 2020.

The extracted information included general characteristics, publication characteristics, research characteristics, and characteristics of the use of the TCS (Table 1).

Articles using TCS scores from primary reports

Almost all of the studies that used the TCS reports as a primary source (n=22) (Table 2), were observational in nature (n=21; 95.5%) and 19 were cross-sectional (86.4%). According to the type of unit of analysis, half were ecological studies and half multilevel. These studies aimed to address the relationship between tobacco control policies and several outcomes, such as the prevalence of preterm births and low birthweight⁷, of hard-core and light smokers⁸, and of smoking in adolescents^9,10; smoking prevalence and quit ratios^4,11; consumption of rolling tobacco, e-cigarettes, and readiness to quit in adults¹²; and risk of lung cancer¹³. Other indicators were smoking in private venues^1,14, self-reported exposure to secondhand smoke (SHS)^15-17, and attitudes towards smoking and tobacco product restrictions^18-21.

Other publications were focused on exploring the association between the price of tobacco products and infant mortality²², the effects of sales restriction laws on adolescents²³, and the factors associated with exposure to tobacco and e-cigarette advertising²⁴. One study assessed the association between smoking prevalence and public sector corruption and other national characteristics²⁵. The main characteristics and results of each article are given in Table 2.

Articles using TCS methodology to compute new scores

As shown in Table 3, ten studies calculated new scores to measure tobacco control policies at a country level in a particular year using the TCS rationale and methodology instead of the original TCS. Most of these studies used data from European countries with a longitudinal design aimed at assessing the association between tobacco control policies and smoking^6,26,27 and socio-economic inequalities outcomes in adolescents²⁸ or adults over time^29-31, or to examine political factors that drive tobacco control policy development³². According to the type of data, these publications were half ecological and half multilevel studies.

Two publications computed scores for nonEuropean countries to monitor their tobacco control policy implementation by using the same rationale and methodology. These publications were aimed at measuring the progress after implementation of the WHO-FCTC in Armenia³³ and providing an overview and comparing the tobacco control progress in Eastern Mediterranean countries³⁴. The main characteristics and results of each study are shown in Table 3.

Main limitations of the TCS mentioned by the studies

Only 11 (34.4%) of all publications included comments on the limitations of using the TCS score as a tool to assess tobacco control policies. The main limitations reported by the studies were that they failed to express the degree to which legislative policies are enforced^6,32,35, except for the smoke-free policies⁴. Another limitation is that the countries’ rankings have only slightly changed over the years (i.e. the UK has remained in the top position from 2007 to 2016)⁴. This low variance across countries may reduce the robustness of the results of the studies¹⁰. Moreover, some studies reported that the information described by the TCS score(s) does not incorporate the most recent national legislation on tobacco control due to its cross-sectional design¹³, potentially underestimating the impact of such policies when using the TCS¹¹.

Finally, among the studies not using data from the original TCS reports, the main limitations were that some policy areas could not be quantified accurately and that some of the policy components assessed by the TCS could not be included³⁴ because of potential error in the measurement of their estimates, and inadequate accuracy and comparability of the data³³.

Interpretation of the results

This is the first attempt to assess all of the available publications that have used the TCS as a means to measure tobacco control policy implementation since it was developed in 2006. In addition, this is the first study to map out the characteristics of the use of the TCS in scientific research, to understand how this tool has been applied despite its original design as a means to advocate for comprehensive tobacco control policies. Therefore, our findings suggest that the TCS has commonly been used as an indicator of the state of tobacco control policies in Europe.

Almost all of the studies assessed tobacco control policies through the total TCS score, and most have used the policy components scores from the primary published reports. The policy components most commonly studied were public smoking bans and tobacco product advertising bans, possibly because they are two of the measures that have been most frequently regulated in Europe since the WHO-FCTC came into force in 2005³⁶.

Another important issue to address is the crosssectional and temporal comparability of the TCS because most of the studies make comparisons across countries and/or over time. Notably, Joossens and Raw⁵ designed the scale to compare tobacco control policies across countries at a particular time point. Thus, the reference values for scores are sustained across each report. However, these scores are not comparable across years because these standards have changed over time (i.e. the weighted average price for cigarettes was €8.5 in 2013 and €10 in 2016, considering the EU average Purchasing Power Standard)^37,38; on the other hand, the scale methodology and scoring system changed between 2007 and 2010³⁹. Consequently, longitudinal studies to ensure temporal comparability between and within countries require adjusting scores to the highest standards by an escalation process and re-calculating scores from the 2005 and 2007 reports using the newest scoring system and methodology.

Importantly, most of the studies with a longitudinal design conducted in Europe have adapted the scale rationale and methods to estimate the level of implementation of tobacco control policies to ensure the temporal comparability and include data about years for which the TCS had not been published^{26,27,29,31,32}. Few non-European countries have adapted the scale as a proxy to monitor the status of tobacco control policies in non-European countries^33,34. Unfortunately, most of these studies did not clearly explain how they adapted the TCS to estimate new score(s) for each policy component. Therefore, new studies should provide a full description of their adaptation process and the potential limitations and strengths not only to ensure its replication, but also to further develop strategies to adapt the TCS to other contexts overseas.

These results highlight that the TCS, regardless of its limitations, has been applied as an objective indicator to measure the strength of the implementation of tobacco control policies at the country level. Other studies have used a total score obtained from summing the scores (from 1 to 5) assigned to each MPOWER policy dimension in the WHO’s Reports on the Global Tobacco Epidemic^40,41. MPOWER’s composite score has some clear advantages over the TCS total score because it is available for all countries, not only European countries, and is comparable over time. However, this proxy also has some disadvantages for research purposes. First, it assigns the same weight to each of the six individual MPOWER scores without taking into consideration that some MPOWER measures have been proven to be more effective than other measures (i.e. taxation). Second, MPOWER’s composite score has a narrower score range than the TCS score (6–29 vs 0–100, respectively), which limits variation across countries and may make it difficult to address variability between countries. Finally, unlike the MPOWER composite score, the TCS score is not affected by the government’s political agenda, as the TCS is built on information from objective databases (i.e. Eurostat) and the Civil Society.

More than 65% of the reviewed publications did not report any limitation of the TCS as a proxy for measuring tobacco control policies. Nonetheless, Joossens and Raw⁵ already reported some of its major limitations, including difficulties in assessing enforcement versus implementation and its critical dependence on tobacco control experts’ judgement when scoring⁵. Therefore, such underreporting of limitations makes it difficult to fully describe the limitations that researchers encounter, which is indispensable to moving forward in the field. Among the articles reporting limitations, most of them highlighted the fact that the TCS score(s) did not measure the enforcement of policies except smokefree policies. In this sense, no previous studies have examined the disparity between the implementation and enforcement of tobacco control policies; however, the TCS being predictive of so many outcomes suggests that the implementation of these policies is a good proxy of enforcement. In addition, some studies have questioned the ability of the TCS to incorporate new policies, such as smoke-free outdoor policies, indicating that the authors of the TCS should discuss how to incorporate these new tobacco control policies and which weight they should have in the scale.

Moreover, our study shows that the TCS has been commonly used in Europe over the last decade, but three research groups from Spain, the Netherlands, and the UK account for more than half of the publications using the TCS for research purposes. This suggests that these groups have led and consolidated the use of this monitoring tool in the tobacco control research field. This is supported by the fact that the publications conducted by these three research groups have received a higher number of citations, including a paper¹³ with 54 citations and another paper²³ with 103 citations (in Web of Science) up to December 2019.

More than half of all publications directed their conclusions towards advocacy for improving tobacco control policies. Therefore, most authors find the TCS useful for linking data to policy action, even though the TCS has been commonly used for research purposes. Therefore, the TCS has not lost its intended original purpose for advocacy, as it was developed to detect areas of improvement within each country and to establish comparisons among countries through a ranking, in order to motivate governments to strengthen their weakest polices⁵.

Our results indicate that, despite its potential limitations and lack of a formal validity assessment, the TCS is a good proxy of the strength of tobacco control policies implementation, or at least the best approximation developed so far. However, the TCS has been used at face-value. No attempts have been made to formally validate the scale. Construct validity of the TCS is a complex issue given the composite structure of the TCS itself, though some dimensions are based on objective data (i.e. price and SHS exposure) from population-based surveys and reports of the European Commission; others are based on the answers of one or two informants to an ad hoc questionnaire (i.e. cessation budget at national level)⁵.

Limitations and strengths

Publication bias is a potential source of error when the units of the investigation are published papers⁴². We searched the available literature in PubMed, the main biomedical database, as well as Web of Science and Google Scholar, and checked all references to identify other articles not published in academic journals. However, the possibility that unpublished manuscripts or other documents addressing the topic of interest may have been missed cannot be ruled out, but it was an a priori decision made by the experienced research team that was composed of tobacco control and policy experts, including the author of the TCS. Under these circumstances, selection (publication) bias seems unlikely to have affected the study.

Other potential limitations of our study are linked to the fact that a high number of the publications analyzed here did not include any comment about the limitations and strengths of using TCS scores as a variable to monitor tobacco control policy implementation in research. This missed reporting has hindered the identification of the main limitations and strengths of this tool for different types of study designs, outcomes, or statistical analysis; therefore, our study may have some missing information.

However, our study is the first to assess all published articles using the TCS as an indicator of tobacco control policy implementation and to characterize its use in tobacco control research, giving a full and comprehensive overview of how and for which purposes the TCS has been employed in previous studies. This study also presents information on how to best use the TCS as described by the authors in the Limitations sections of the publications. Thus, our study presents the lessons learned from previous research, creating an opportunity for researchers to plan to use the TCS to improve the quality of future studies.