what is the outcome variable in research

Research Variables 101

Independent variables, dependent variables, control variables and more

By: Derek Jansen (MBA) | Expert Reviewed By: Kerryn Warren (PhD) | January 2023

If you’re new to the world of research, especially scientific research, you’re bound to run into the concept of variables , sooner or later. If you’re feeling a little confused, don’t worry – you’re not the only one! Independent variables, dependent variables, confounding variables – it’s a lot of jargon. In this post, we’ll unpack the terminology surrounding research variables using straightforward language and loads of examples .

Overview: Variables In Research

What (exactly) is a variable?

The simplest way to understand a variable is as any characteristic or attribute that can experience change or vary over time or context – hence the name “variable”. For example, the dosage of a particular medicine could be classified as a variable, as the amount can vary (i.e., a higher dose or a lower dose). Similarly, gender, age or ethnicity could be considered demographic variables, because each person varies in these respects.

Within research, especially scientific research, variables form the foundation of studies, as researchers are often interested in how one variable impacts another, and the relationships between different variables. For example:

• How someone’s age impacts their sleep quality
• How different teaching methods impact learning outcomes
• How diet impacts weight (gain or loss)

As you can see, variables are often used to explain relationships between different elements and phenomena. In scientific studies, especially experimental studies, the objective is often to understand the causal relationships between variables. In other words, the role of cause and effect between variables. This is achieved by manipulating certain variables while controlling others – and then observing the outcome. But, we’ll get into that a little later…

The “Big 3” Variables

Variables can be a little intimidating for new researchers because there are a wide variety of variables, and oftentimes, there are multiple labels for the same thing. To lay a firm foundation, we’ll first look at the three main types of variables, namely:

• Independent variables (IV)
• Dependant variables (DV)
• Control variables

What is an independent variable?

Simply put, the independent variable is the “ cause ” in the relationship between two (or more) variables. In other words, when the independent variable changes, it has an impact on another variable.

For example:

• Increasing the dosage of a medication (Variable A) could result in better (or worse) health outcomes for a patient (Variable B)
• Changing a teaching method (Variable A) could impact the test scores that students earn in a standardised test (Variable B)
• Varying one’s diet (Variable A) could result in weight loss or gain (Variable B).

It’s useful to know that independent variables can go by a few different names, including, explanatory variables (because they explain an event or outcome) and predictor variables (because they predict the value of another variable). Terminology aside though, the most important takeaway is that independent variables are assumed to be the “cause” in any cause-effect relationship. As you can imagine, these types of variables are of major interest to researchers, as many studies seek to understand the causal factors behind a phenomenon.

Need a helping hand?

What is a dependent variable?

While the independent variable is the “ cause ”, the dependent variable is the “ effect ” – or rather, the affected variable . In other words, the dependent variable is the variable that is assumed to change as a result of a change in the independent variable.

Keeping with the previous example, let’s look at some dependent variables in action:

• Health outcomes (DV) could be impacted by dosage changes of a medication (IV)
• Students’ scores (DV) could be impacted by teaching methods (IV)
• Weight gain or loss (DV) could be impacted by diet (IV)

In scientific studies, researchers will typically pay very close attention to the dependent variable (or variables), carefully measuring any changes in response to hypothesised independent variables. This can be tricky in practice, as it’s not always easy to reliably measure specific phenomena or outcomes – or to be certain that the actual cause of the change is in fact the independent variable.

As the adage goes, correlation is not causation . In other words, just because two variables have a relationship doesn’t mean that it’s a causal relationship – they may just happen to vary together. For example, you could find a correlation between the number of people who own a certain brand of car and the number of people who have a certain type of job. Just because the number of people who own that brand of car and the number of people who have that type of job is correlated, it doesn’t mean that owning that brand of car causes someone to have that type of job or vice versa. The correlation could, for example, be caused by another factor such as income level or age group, which would affect both car ownership and job type.

To confidently establish a causal relationship between an independent variable and a dependent variable (i.e., X causes Y), you’ll typically need an experimental design , where you have complete control over the environmen t and the variables of interest. But even so, this doesn’t always translate into the “real world”. Simply put, what happens in the lab sometimes stays in the lab!

As an alternative to pure experimental research, correlational or “ quasi-experimental ” research (where the researcher cannot manipulate or change variables) can be done on a much larger scale more easily, allowing one to understand specific relationships in the real world. These types of studies also assume some causality between independent and dependent variables, but it’s not always clear. So, if you go this route, you need to be cautious in terms of how you describe the impact and causality between variables and be sure to acknowledge any limitations in your own research.

What is a control variable?

In an experimental design, a control variable (or controlled variable) is a variable that is intentionally held constant to ensure it doesn’t have an influence on any other variables. As a result, this variable remains unchanged throughout the course of the study. In other words, it’s a variable that’s not allowed to vary – tough life 🙂

As we mentioned earlier, one of the major challenges in identifying and measuring causal relationships is that it’s difficult to isolate the impact of variables other than the independent variable. Simply put, there’s always a risk that there are factors beyond the ones you’re specifically looking at that might be impacting the results of your study. So, to minimise the risk of this, researchers will attempt (as best possible) to hold other variables constant . These factors are then considered control variables.

Some examples of variables that you may need to control include:

• Temperature
• Time of day
• Noise or distractions

Which specific variables need to be controlled for will vary tremendously depending on the research project at hand, so there’s no generic list of control variables to consult. As a researcher, you’ll need to think carefully about all the factors that could vary within your research context and then consider how you’ll go about controlling them. A good starting point is to look at previous studies similar to yours and pay close attention to which variables they controlled for.

Of course, you won’t always be able to control every possible variable, and so, in many cases, you’ll just have to acknowledge their potential impact and account for them in the conclusions you draw. Every study has its limitations , so don’t get fixated or discouraged by troublesome variables. Nevertheless, always think carefully about the factors beyond what you’re focusing on – don’t make assumptions!

Other types of variables

As we mentioned, independent, dependent and control variables are the most common variables you’ll come across in your research, but they’re certainly not the only ones you need to be aware of. Next, we’ll look at a few “secondary” variables that you need to keep in mind as you design your research.

• Moderating variables
• Mediating variables
• Confounding variables
• Latent variables

Let’s jump into it…

What is a moderating variable?

A moderating variable is a variable that influences the strength or direction of the relationship between an independent variable and a dependent variable. In other words, moderating variables affect how much (or how little) the IV affects the DV, or whether the IV has a positive or negative relationship with the DV (i.e., moves in the same or opposite direction).

For example, in a study about the effects of sleep deprivation on academic performance, gender could be used as a moderating variable to see if there are any differences in how men and women respond to a lack of sleep. In such a case, one may find that gender has an influence on how much students’ scores suffer when they’re deprived of sleep.

It’s important to note that while moderators can have an influence on outcomes , they don’t necessarily cause them ; rather they modify or “moderate” existing relationships between other variables. This means that it’s possible for two different groups with similar characteristics, but different levels of moderation, to experience very different results from the same experiment or study design.

What is a mediating variable?

Mediating variables are often used to explain the relationship between the independent and dependent variable (s). For example, if you were researching the effects of age on job satisfaction, then education level could be considered a mediating variable, as it may explain why older people have higher job satisfaction than younger people – they may have more experience or better qualifications, which lead to greater job satisfaction.

Mediating variables also help researchers understand how different factors interact with each other to influence outcomes. For instance, if you wanted to study the effect of stress on academic performance, then coping strategies might act as a mediating factor by influencing both stress levels and academic performance simultaneously. For example, students who use effective coping strategies might be less stressed but also perform better academically due to their improved mental state.

In addition, mediating variables can provide insight into causal relationships between two variables by helping researchers determine whether changes in one factor directly cause changes in another – or whether there is an indirect relationship between them mediated by some third factor(s). For instance, if you wanted to investigate the impact of parental involvement on student achievement, you would need to consider family dynamics as a potential mediator, since it could influence both parental involvement and student achievement simultaneously.

What is a confounding variable?

A confounding variable (also known as a third variable or lurking variable ) is an extraneous factor that can influence the relationship between two variables being studied. Specifically, for a variable to be considered a confounding variable, it needs to meet two criteria:

• It must be correlated with the independent variable (this can be causal or not)
• It must have a causal impact on the dependent variable (i.e., influence the DV)

Some common examples of confounding variables include demographic factors such as gender, ethnicity, socioeconomic status, age, education level, and health status. In addition to these, there are also environmental factors to consider. For example, air pollution could confound the impact of the variables of interest in a study investigating health outcomes.

Naturally, it’s important to identify as many confounding variables as possible when conducting your research, as they can heavily distort the results and lead you to draw incorrect conclusions . So, always think carefully about what factors may have a confounding effect on your variables of interest and try to manage these as best you can.

What is a latent variable?

Latent variables are unobservable factors that can influence the behaviour of individuals and explain certain outcomes within a study. They’re also known as hidden or underlying variables , and what makes them rather tricky is that they can’t be directly observed or measured . Instead, latent variables must be inferred from other observable data points such as responses to surveys or experiments.

For example, in a study of mental health, the variable “resilience” could be considered a latent variable. It can’t be directly measured , but it can be inferred from measures of mental health symptoms, stress, and coping mechanisms. The same applies to a lot of concepts we encounter every day – for example:

• Emotional intelligence
• Quality of life
• Business confidence
• Ease of use

One way in which we overcome the challenge of measuring the immeasurable is latent variable models (LVMs). An LVM is a type of statistical model that describes a relationship between observed variables and one or more unobserved (latent) variables. These models allow researchers to uncover patterns in their data which may not have been visible before, thanks to their complexity and interrelatedness with other variables. Those patterns can then inform hypotheses about cause-and-effect relationships among those same variables which were previously unknown prior to running the LVM. Powerful stuff, we say!

Let’s recap

In the world of scientific research, there’s no shortage of variable types, some of which have multiple names and some of which overlap with each other. In this post, we’ve covered some of the popular ones, but remember that this is not an exhaustive list .

To recap, we’ve explored:

• Independent variables (the “cause”)
• Dependent variables (the “effect”)
• Control variables (the variable that’s not allowed to vary)

If you’re still feeling a bit lost and need a helping hand with your research project, check out our 1-on-1 coaching service , where we guide you through each step of the research journey. Also, be sure to check out our free dissertation writing course and our collection of free, fully-editable chapter templates .

Psst... there’s more!

This post was based on one of our popular Research Bootcamps . If you're working on a research project, you'll definitely want to check this out ...

You Might Also Like:

Very informative, concise and helpful. Thank you

Helping information.Thanks

practical and well-demonstrated

Very helpful and insightful

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

User Preferences

Content preview.

Arcu felis bibendum ut tristique et egestas quis:

• Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris
• Duis aute irure dolor in reprehenderit in voluptate
• Excepteur sint occaecat cupidatat non proident

Keyboard Shortcuts

1.1.2 - explanatory & response variables.

In some research studies one variable is used to predict or explain differences in another variable. In those cases, the  explanatory variable  is used to predict or explain differences in the  response variable . In an experimental study, the explanatory variable is the variable that is manipulated by the researcher. 

Also known as the independent  or  predictor variable , it explains variations in the response variable; in an experimental study, it is manipulated by the researcher

Also known as the  dependent  or  outcome variable,  its value is predicted or its variation is explained by the explanatory variable; in an experimental study, this is the outcome that is measured following manipulation of the explanatory variable

Example: Panda Fertility Treatments Section  

A team of veterinarians wants to compare the effectiveness of two fertility treatments for pandas in captivity. The two treatments are in-vitro fertilization and male fertility medications. This experiment has one  explanatory variable : type of fertility treatment. The  response variable  is a measure of fertility rate.

Example: Public Speaking Approaches Section  

A public speaking teacher has developed a new lesson that she believes decreases student anxiety in public speaking situations more than the old lesson. She designs an experiment to test if her new lesson works better than the old lesson. Public speaking students are randomly assigned to receive either the new or old lesson; their anxiety levels during a variety of public speaking experiences are measured.  This experiment has one  explanatory variable : the lesson received. The  response variable  is anxiety level.

Example: Coffee Bean Origin Section  

A researcher believes that the origin of the beans used to make a cup of coffee affects hyperactivity. He wants to compare coffee from three different regions: Africa, South America, and Mexico. The  explanatory variable is the origin of coffee bean; this has three levels: Africa, South America, and Mexico. The  response variable  is hyperactivity level.

Example: Height & Age Section  

A group of middle school students wants to know if they can use height to predict age. They take a random sample of 50 people at their school, both students and teachers, and record each individual's height and age. This is an observational study. The students want to use height to predict age so the  explanatory variable  is height and the  response variable  is age.

Example: Grade & Height Section  

Research question:  Do fourth graders tend to be taller than third graders?

This is an observational study. The researcher wants to use grade level to explain differences in height. The  explanatory variable  is grade level. The  response variable  is height. 

COMMON MISTEAKS MISTAKES IN USING STATISTICS: Spotting and Avoiding Them

Glossary      blog, choosing an outcome 1 variable, example 1: how to measure "big", example 2: how to measure "unemployment rate".

• Do not assume you understand what a measure is just because the name makes sense to you. Be sure to find and read the definition carefully; it may not be what you think.
• Be especially careful when making comparisons. The same term might be used differently by different authors or in different places. For example, different countries have different definitions of unemployment rate. (See http://www.bls.gov/fls/flsfaqs.htm#laborforcedefinitions )

Example 3: What is a good outcome variable for deciding whether cancer treatment in a country has been improving?

Example 4: what is a good outcome variable for answering the question, "do males or females suffer more traffic fatalities", example 5: what is a good outcome variable for research on the effect of medication on bone fractures  , statistical considerations.

13 Predictor and Outcome Variable Examples

Dave Cornell (PhD)

Dr. Cornell has worked in education for more than 20 years. His work has involved designing teacher certification for Trinity College in London and in-service training for state governments in the United States. He has trained kindergarten teachers in 8 countries and helped businessmen and women open baby centers and kindergartens in 3 countries.

Learn about our Editorial Process

Chris Drew (PhD)

This article was peer-reviewed and edited by Chris Drew (PhD). The review process on Helpful Professor involves having a PhD level expert fact check, edit, and contribute to articles. Reviewers ensure all content reflects expert academic consensus and is backed up with reference to academic studies. Dr. Drew has published over 20 academic articles in scholarly journals. He is the former editor of the Journal of Learning Development in Higher Education and holds a PhD in Education from ACU.

A predictor variable is used to predict the occurrence and/or level of another variable, called the outcome variable.

A researcher will measure both variables in a scientific study and then use statistical software to determine if the predictor variable is associated with the outcome variable. If there is a strong correlation, we say the predictor variable has high predictive validity .

This methodology is often used in epidemiological research. Researchers will measure both variables in a given population and then determine the degree of association between the predictor and outcome variable.

This allows scientists to examine the connection between many meaningful variables, such as exercise and health or personality type and depression, just to give a few examples.

Although this type of research can provide significant insights that help us understand a phenomenon, we cannot say that the predictor valuable causes the outcome variable.

In order to use the term ‘cause and effect’, the researcher must be able to control and manipulate the level of a variable and then observe the changes in the other variable.

Definition of Predictor and Outcome Variables

In reality, many variables usually affect the outcome variable. So, researchers will measure numerous predictor variables in the population under study and then determine the degree of association that each one has with the outcome variable.

It sounds a bit complicated, but fortunately, the use of a statistical technique called multiple regression analysis simplifies the process.

As long as the variables are measured accurately and the population size is large, the software will be able to determine which of the predictor variables are associated with the outcome variable and the degree of association.

Not all predictors will have an equal influence on the outcome variable. Some may have a very small impact, some may have a substantial impact, and others may have no impact at all.

Predictor and outcome are not to be confused with independent and dependent variables .

Examples of Predictor and Outcome Variables

1. diet and health.

Does the food you eat have any impact on your physical health? This is a question that a lot of people want to know the answer to.

Many of us have very poor diets, with lots of fast food and salty snacks. Other people, however, almost never make a run through the drive-thru, and consume mostly fruits and veggies.

Thankfully, epidemiological research can give us a relatively straightforward answer. First, researchers measure the quality of diet of each person in a large population.

So, they will track how much fast food and fruits and veggies people consume. There are a lot of different ways to measure this.

Secondly, researchers will measure some aspects of health. This could involve checking cholesterol levels, for example. There are a lot of different ways to measure health. The final step is to input all of the data into the statistical software program and perform the regression analysis to see the results.

Quality of diet is the predictor variable, and health is the outcome variable.

2. Noise Pollution and IQ

One scientist speculates that living in a noisy environment will affect a person’s ability to concentrate, which will then affect their mental acuity and subsequent cognitive development .

So, they decide to conduct a study examining the relationship between noise pollution and IQ.

First, they travel through lots of different neighborhoods and use a sound level meter to assess noise pollution. Some neighborhoods are in the suburbs, and some are near busy highways or construction sites.

Next, they collect data on SAT scores of the children living in those neighborhoods.

They then conduct a regression analysis to determine the connection between the sound level meter data and the SAT scores.

In this example, the predictor variable is the sound levels, and the outcome variable is the SAT scores.

Surprisingly, the results revealed an inverse relationship between noise and SAT scores. That is, the more noise in the environment the higher the SAT score. Any idea why?

3. Family Income and Achievement Test Scores

In this study, sociologists conducted a study examining the relationship between how much income a family has and the achievement test scores of their children.

The researchers collected data from schools on the achievement test scores of hundreds of students and then estimated the household income of the families based on the occupation of the parents.

The results revealed a strong relationship between family income and test scores, such that the higher the family income, the higher the test score of the child.

In this example, family income is the predictor variable, and test score is the outcome variable.

4. Parental Utterances and Children’s Vocabulary

A team of child psychologist is interested in the impact of how much parents talk to their child and that child’s verbal skills.

So, they design a study that involves observing families in the home environment. They randomly choose 50 families to study that live nearby.

A research assistant visits each family, records, and later counts the number of utterances spoken by the mother directed at their only child.

On a different occasion, a second research assistant administers a verbal skills test to every child. Yes, this type of study takes a lot of time.

The regression analysis reveals a direct relationship between the number of utterances from the mother and the child’s verbal skills test score. The more utterances, the higher the score.

In this example, the predictor variable is the number of utterances directed at the child, and the outcome variable is the child’s verbal skills test score.

5. Video Games and Aggressiveness

The debate about the effects of TV violence and video games has been raging for nearly 70 years. There have been hundreds, maybe even thousands of studies conducted on the issue.

One type of study involves assessing how frequently a group of people play certain video games and then tracking their level of aggressiveness over a period of time.

Of course, there are other factors involved in whether a person is aggressive or not, so the researchers might assess those variables as well.

In this type of study, the predictor variable is the frequency of playing video games, and the outcome variable is the level of aggressiveness.

6. Chemicals in Food Products and Puberty

In many countries, farmers may inject various antibiotics and growth hormones into their cattle to ward off infection and increase body mass and milk production.

Unfortunately, those chemicals do not disappear once the food hits the supermarket shelves. Some parents, educators, and food scientists began to notice an association between these agricultural practices and the onset of puberty in young children.

Numerous scientific studies were conducted examining the relationship between these practices and puberty.

So, the researchers studied the relationship between the predictor variable (chemicals in food) and the outcome variable (onset of puberty).

7. Full Moon and Craziness

Who hasn’t heard that a full moon brings out the crazies? A lot of people have theorized that when the moon is full, people get a little bit wild and uninhibited.

That can lead to people doing things they would not normally do.

To put this theory to the test, a group of criminologists decides to examine the police records of numerous large cities and compare that with the lunar cycle.

The researchers input all of the data into a stats program to examine the degree of association between police incidents and the moon.

In this study, the lunar cycle is the predictor variable, and contravention of the law is the outcome variable.  

8. Testosterone and Leadership Style

There are many types of leadership styles. Some leaders are very people-oriented and try to help their employees prosper and feel good about their jobs.

Other leaders are more task-driven and prefer to clearly define objectives, set deadlines, and push their staff to work hard.

To examine the relationship between leadership style and testosterone, a researcher first administers a questionnaire to hundreds of employees in several types of companies. The questionnaire asks the employees to describe the leadership style of their primary supervisor.

At the same time, the researcher also collects data on the testosterone levels of those supervisors and matches them with the questionnaire data.

By examining the association between the two, it will be possible to determine if there is a link between leadership style and testosterone.

The predictor variable is testosterone, and the outcome variable is leadership style.  

9. Personality Type and Driver Safety

A national bus company wants to hire the safest drivers possible. Fewer accidents mean passengers will be safe and their insurance rates will be lower. 

So, the HR staff begin collecting data on the safety records of their drivers over the last 3 years. At the same time, they administer a personality inventory that assesses Type A and Type B personalities.

The Type A personality is intense, impatient, and highly competitive. The Type B personality is easygoing and relaxed. People have varying levels of each type.

The HR department wants to know if there is a relationship between personality type (A or B) and accidents among their drivers.

The predictor variable is personality type, and the outcome variable is the number of accidents.

10. Vitamins and Health

Americans take a lot of vitamins. However, there is some debate about whether vitamins actually do anything to improve health.

There are so many factors that affect health, will taking a daily supplement really count?

So, a group of small vitamin companies pull their resources and hire an outside consulting firm to conduct a large-scale scientific study.

The firm randomly selects thousands of people from throughout the country to participate in the study. The people selected come from a wide range of SES backgrounds, ethnicities, and ages.

Each person is asked to go to a nearby hospital and have a basic health screening that includes cholesterol and blood pressure. They also respond to a questionnaire that asks if they take a multi-vitamin, how many and how often.

The consulting firm then compares the degree of association between multi-vitamins and health.

Multi-vitamin use is the predictor variable, and health is the outcome variable.

11. Automobiles and Climate Change

A group of climatologists has received funding from the EU to conduct a large-scale study on climate change.

The researchers collect data on a wide range of variables that are suspected of affecting the climate. Some of those variables include automobile production, industrial output, size of cattle herds, and deforestation, just to name a few.

The researchers proceed by gathering the data beginning with the 1970s all the way to the current year. They also collect data on yearly temperature fluctuations.

Once all the data is collected, it is put into a stats program, and a few minutes later, the results are revealed.

In this example, there are many predictor variables, such as automobile production, and one primary outcome variable (yearly temperature fluctuations).

12. Smartphone Use and Eye Strain

If you’ve ever noticed, people spend a lot of time looking at their smartphones.

When they are reading, when they are waiting in line, in bed at night, and even when walking from point A to point B.

Many optometrists are concerned that all of this screen time is doing harm to people’s eyesight. So, they decide to conduct a study.

Fortunately, they all work for a nationwide optometry company with offices located in Wal-Marts.

When patients come into their office, they give each one a standard eye exam. They also put a question on the in-take form asking each person to estimate how many hours a day they spend looking at their smartphone screen. 

Then they examine the relation between screen-time usage and the results of the eye exams.

In this study, the predictor variable is screen-time, and the outcome variable is the eye-exam results.

13. Soil Composition and Agricultural Yields

Although farming looks easy, it can be a very scientific enterprise. Agriculturalists study the composition of soil to help determine what type of food will grow best.

Today, they know a lot about which soil nutrients affect the growth of different plant varieties because there have been decades of studies.

The research involves collecting soil samples, measuring crop yields, and then examining the association between the two.

For example, scientists will measure the pH levels, mineral composition, as well as water and air content over many acres of land and relate that to the amount harvested of a particular crop (e.g., corn).

In this example, there are numerous predictor variables, all of which have some effect on crop growth, which is the outcome variable.

Even though there are so many variables to consider, the regression analysis will be able to tell us how important each one is in predicting the outcome variable.

There can be a lot of reasons why something happens. More often than not, nothing happens as a result of just one factor. Our physical health, climate change, and a person’s level of aggressiveness are all the result of numerous factors.

Fortunately for science, there is a brilliant way of determining which factors are connected to a phenomenon and how strong is each and every one of them.

By collecting data on a predictor variable (or variables) and then examining the association with the outcome variable, we can gain valuable insights into just about any subject matter we wish to study.

Ferguson, C. J., & Kilburn, J. (2010). Much ado about nothing: The misestimation and overinterpretation of violent video game effects in Eastern and Western nations: Comment on Anderson et al. (2010). Psychological Bulletin, 136 (2), 174–178. https://doi.org/10.1037/a0018566

Ferguson, C. J., San Miguel, C., Garza, A., & Jerabeck, J. M. (2012). A longitudinal test of video game violence influences on dating and aggression: A 3-year longitudinal study of adolescents, Journal of Psychiatric Research, 46 (2), 141-146. https://doi.org/10.1016/j.jpsychires.2011.10.014

Gordon, R. (2015). Regression Analysis for the Social Sciences (2 nd ed). New York: Routledge.

Hoff, E. (2003). The specificity of environmental influence: Socioeconomic status affects early vocabulary development via maternal speech. Child Development, 74 (5), 1368–1378.

Lopez-Rodriguez, D., Franssen, D., Heger, S., & Parent, AS. (2011). Endocrine-disrupting chemicals and their effects on puberty. Best Practice & Research Clinical Endocrinology & Metabolism, 35 (5), 101579. https://doi.org/10.1016/j.beem.2021.101579

Man, A., Li, H., & Xia, N. (2020). Impact of lifestyles (Diet and Exercise) on vascular health: Oxidative stress and endothelial function. Oxidative Medicine and Cellular Longevity , 1496462. https://doi.org/10.1155/2020/1496462

Thompson, R., Smith, R. B., Karim, Y. B., Shen, C., Drummond, K., Teng, C., & Toledano, M. B. (2022). Noise pollution and human cognition: An updated systematic review and meta-analysis of recent evidence. Environment International , 158 , 106905.

• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 23 Achieved Status Examples
• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 25 Defense Mechanisms Examples
• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 15 Theory of Planned Behavior Examples
• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 18 Adaptive Behavior Examples

• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd-2/ 23 Achieved Status Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd-2/ 15 Ableism Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd-2/ 25 Defense Mechanisms Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd-2/ 15 Theory of Planned Behavior Examples

1 thought on “13 Predictor and Outcome Variable Examples”

If I want to undertake an interventional study where I measure the Knowledge, attitudes and practices of adolescents in 3 key sexual and reproductive areas. And their parents’ acceptance of ASRH education for their children, and their misconceptions of ASRH. And then I introduce both children and parents to ASRH education. Then I do an end line to look for improvement in the adolescent’s KAP in those 3 areas, and an increased acceptance of ASRH education among parents, what is my predictor variable and outcome variable?

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

The primary outcome measure and its importance in clinical trials

Affiliation.

• 1 Department of Psychopharmacology, National Institute of Mental Health and Neurosciences, Bangalore, India [email protected].
• PMID: 26528658
• DOI: 10.4088/JCP.15f10377

The primary outcome measure is the outcome that an investigator considers to be the most important among the many outcomes that are to be examined in the study. The primary outcome needs to be defined at the time the study is designed. There are 2 reasons for this: it reduces the risk of false-positive errors resulting from the statistical testing of many outcomes, and it reduces the risk of a false-negative error by providing the basis for the estimation of the sample size necessary for an adequately powered study. This article discusses the setting of the primary outcome measure, the need for it, the increased risk of false-positive and false-negative errors in secondary outcome results, how to regard articles that do not state the primary outcome, how to interpret results when secondary outcomes are statistically significant but not the primary outcome, and limitations of the concept of a primary outcome measure in clinical trial research.

© Copyright 2015 Physicians Postgraduate Press, Inc.

PubMed Disclaimer

Similar articles

• Trussed in evidence? Ambiguities at the interface between clinical evidence and clinical practice. Healy D. Healy D. Transcult Psychiatry. 2009 Mar;46(1):16-37. doi: 10.1177/1363461509102285. Transcult Psychiatry. 2009. PMID: 19293278 Review.
• Randomized controlled trials and psychotherapy research. Rifkin A. Rifkin A. Am J Psychiatry. 2007 Jan;164(1):7-8. doi: 10.1176/ajp.2007.164.1.7. Am J Psychiatry. 2007. PMID: 17202535 No abstract available.
• Psychopharmacology: the need for effectiveness trials to inform evidence-based psychiatric practice. Lagomasino IT, Dwight-Johnson M, Simpson GM. Lagomasino IT, et al. Psychiatr Serv. 2005 Jun;56(6):649-51. doi: 10.1176/appi.ps.56.6.649. Psychiatr Serv. 2005. PMID: 15939939 No abstract available.
• Stepped care: doing more with less? Davison GC. Davison GC. J Consult Clin Psychol. 2000 Aug;68(4):580-5. J Consult Clin Psychol. 2000. PMID: 10965633 Review.
• Traditional and alternative research designs and methods in clinical pediatric psychopharmacology. Fava M. Fava M. J Am Acad Child Adolesc Psychiatry. 1996 Oct;35(10):1292-303. doi: 10.1097/00004583-199610000-00016. J Am Acad Child Adolesc Psychiatry. 1996. PMID: 8885583 Review.
• Mendelian randomization as a tool to inform drug development using human genetics. Daghlas I, Gill D. Daghlas I, et al. Camb Prism Precis Med. 2023 Feb 8;1:e16. doi: 10.1017/pcm.2023.5. eCollection 2023. Camb Prism Precis Med. 2023. PMID: 38550933 Free PMC article. Review.
• Types of Analysis: Planned (prespecified) vs Post Hoc, Primary vs Secondary, Hypothesis-driven vs Exploratory, Subgroup and Sensitivity, and Others. Andrade C. Andrade C. Indian J Psychol Med. 2023 Nov;45(6):640-641. doi: 10.1177/02537176231216842. Epub 2023 Nov 22. Indian J Psychol Med. 2023. PMID: 38545527 Free PMC article.
• Selection of indicators reporting response rate in pharmaceutical trials for systemic lupus erythematosus: preference and relative sensitivity. Tian J, Kang S, Zhang D, Huang Y, Yao X, Zhao M, Lu Q. Tian J, et al. Lupus Sci Med. 2023 Oct;10(2):e000942. doi: 10.1136/lupus-2023-000942. Lupus Sci Med. 2023. PMID: 37798046 Free PMC article.
• Completeness and consistency of primary outcome reporting in COVID-19 publications in the early pandemic phase: a descriptive study. Stoll M, Lindner S, Marquardt B, Salholz-Hillel M, DeVito NJ, Klemperer D, Lieb K. Stoll M, et al. BMC Med Res Methodol. 2023 Jul 29;23(1):173. doi: 10.1186/s12874-023-01991-9. BMC Med Res Methodol. 2023. PMID: 37516878 Free PMC article.
• Risks associated with over-analysis of data. Andrade C, D'Cruz M, Sharma N, Vadlamani N. Andrade C, et al. Indian J Med Res. 2022 Jul;156(1):155. doi: 10.4103/ijmr.IJMR_558_20. Indian J Med Res. 2022. PMID: 36510908 Free PMC article. No abstract available.

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Physicians Postgraduate Press, Inc.
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Independent vs. Dependent Variables | Definition & Examples

Independent vs. Dependent Variables | Definition & Examples

Published on February 3, 2022 by Pritha Bhandari . Revised on June 22, 2023.

In research, variables are any characteristics that can take on different values, such as height, age, temperature, or test scores.

Researchers often manipulate or measure independent and dependent variables in studies to test cause-and-effect relationships.

• The independent variable is the cause. Its value is independent of other variables in your study.
• The dependent variable is the effect. Its value depends on changes in the independent variable.

Your independent variable is the temperature of the room. You vary the room temperature by making it cooler for half the participants, and warmer for the other half.

Table of contents

What is an independent variable, types of independent variables, what is a dependent variable, identifying independent vs. dependent variables, independent and dependent variables in research, visualizing independent and dependent variables, other interesting articles, frequently asked questions about independent and dependent variables.

An independent variable is the variable you manipulate or vary in an experimental study to explore its effects. It’s called “independent” because it’s not influenced by any other variables in the study.

Independent variables are also called:

• Explanatory variables (they explain an event or outcome)
• Predictor variables (they can be used to predict the value of a dependent variable)
• Right-hand-side variables (they appear on the right-hand side of a regression equation).

These terms are especially used in statistics , where you estimate the extent to which an independent variable change can explain or predict changes in the dependent variable.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

There are two main types of independent variables.

• Experimental independent variables can be directly manipulated by researchers.
• Subject variables cannot be manipulated by researchers, but they can be used to group research subjects categorically.

Experimental variables

In experiments, you manipulate independent variables directly to see how they affect your dependent variable. The independent variable is usually applied at different levels to see how the outcomes differ.

You can apply just two levels in order to find out if an independent variable has an effect at all.

You can also apply multiple levels to find out how the independent variable affects the dependent variable.

You have three independent variable levels, and each group gets a different level of treatment.

You randomly assign your patients to one of the three groups:

• A low-dose experimental group
• A high-dose experimental group
• A placebo group (to research a possible placebo effect )

A true experiment requires you to randomly assign different levels of an independent variable to your participants.

Random assignment helps you control participant characteristics, so that they don’t affect your experimental results. This helps you to have confidence that your dependent variable results come solely from the independent variable manipulation.

Subject variables

Subject variables are characteristics that vary across participants, and they can’t be manipulated by researchers. For example, gender identity, ethnicity, race, income, and education are all important subject variables that social researchers treat as independent variables.

It’s not possible to randomly assign these to participants, since these are characteristics of already existing groups. Instead, you can create a research design where you compare the outcomes of groups of participants with characteristics. This is a quasi-experimental design because there’s no random assignment. Note that any research methods that use non-random assignment are at risk for research biases like selection bias and sampling bias .

Your independent variable is a subject variable, namely the gender identity of the participants. You have three groups: men, women and other.

Your dependent variable is the brain activity response to hearing infant cries. You record brain activity with fMRI scans when participants hear infant cries without their awareness.

A dependent variable is the variable that changes as a result of the independent variable manipulation. It’s the outcome you’re interested in measuring, and it “depends” on your independent variable.

In statistics , dependent variables are also called:

• Response variables (they respond to a change in another variable)
• Outcome variables (they represent the outcome you want to measure)
• Left-hand-side variables (they appear on the left-hand side of a regression equation)

The dependent variable is what you record after you’ve manipulated the independent variable. You use this measurement data to check whether and to what extent your independent variable influences the dependent variable by conducting statistical analyses.

Based on your findings, you can estimate the degree to which your independent variable variation drives changes in your dependent variable. You can also predict how much your dependent variable will change as a result of variation in the independent variable.

Distinguishing between independent and dependent variables can be tricky when designing a complex study or reading an academic research paper .

A dependent variable from one study can be the independent variable in another study, so it’s important to pay attention to research design .

Here are some tips for identifying each variable type.

Recognizing independent variables

Use this list of questions to check whether you’re dealing with an independent variable:

• Is the variable manipulated, controlled, or used as a subject grouping method by the researcher?
• Does this variable come before the other variable in time?
• Is the researcher trying to understand whether or how this variable affects another variable?

Recognizing dependent variables

Check whether you’re dealing with a dependent variable:

• Is this variable measured as an outcome of the study?
• Is this variable dependent on another variable in the study?
• Does this variable get measured only after other variables are altered?

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Independent and dependent variables are generally used in experimental and quasi-experimental research.

Here are some examples of research questions and corresponding independent and dependent variables.

For experimental data, you analyze your results by generating descriptive statistics and visualizing your findings. Then, you select an appropriate statistical test to test your hypothesis .

The type of test is determined by:

• your variable types
• level of measurement
• number of independent variable levels.

You’ll often use t tests or ANOVAs to analyze your data and answer your research questions.

In quantitative research , it’s good practice to use charts or graphs to visualize the results of studies. Generally, the independent variable goes on the x -axis (horizontal) and the dependent variable on the y -axis (vertical).

The type of visualization you use depends on the variable types in your research questions:

• A bar chart is ideal when you have a categorical independent variable.
• A scatter plot or line graph is best when your independent and dependent variables are both quantitative.

To inspect your data, you place your independent variable of treatment level on the x -axis and the dependent variable of blood pressure on the y -axis.

You plot bars for each treatment group before and after the treatment to show the difference in blood pressure.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

An independent variable is the variable you manipulate, control, or vary in an experimental study to explore its effects. It’s called “independent” because it’s not influenced by any other variables in the study.

A dependent variable is what changes as a result of the independent variable manipulation in experiments . It’s what you’re interested in measuring, and it “depends” on your independent variable.

In statistics, dependent variables are also called:

Determining cause and effect is one of the most important parts of scientific research. It’s essential to know which is the cause – the independent variable – and which is the effect – the dependent variable.

You want to find out how blood sugar levels are affected by drinking diet soda and regular soda, so you conduct an experiment .

• The type of soda – diet or regular – is the independent variable .
• The level of blood sugar that you measure is the dependent variable – it changes depending on the type of soda.

No. The value of a dependent variable depends on an independent variable, so a variable cannot be both independent and dependent at the same time. It must be either the cause or the effect, not both!

Yes, but including more than one of either type requires multiple research questions .

For example, if you are interested in the effect of a diet on health, you can use multiple measures of health: blood sugar, blood pressure, weight, pulse, and many more. Each of these is its own dependent variable with its own research question.

You could also choose to look at the effect of exercise levels as well as diet, or even the additional effect of the two combined. Each of these is a separate independent variable .

To ensure the internal validity of an experiment , you should only change one independent variable at a time.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bhandari, P. (2023, June 22). Independent vs. Dependent Variables | Definition & Examples. Scribbr. Retrieved June 21, 2024, from https://www.scribbr.com/methodology/independent-and-dependent-variables/

Is this article helpful?

Pritha Bhandari

Other students also liked, guide to experimental design | overview, steps, & examples, explanatory and response variables | definitions & examples, confounding variables | definition, examples & controls, what is your plagiarism score.

Variables in Research | Types, Definiton & Examples

Introduction

What is a variable, what are the 5 types of variables in research, other variables in research.

Variables are fundamental components of research that allow for the measurement and analysis of data. They can be defined as characteristics or properties that can take on different values. In research design , understanding the types of variables and their roles is crucial for developing hypotheses , designing methods , and interpreting results .

This article outlines the the types of variables in research, including their definitions and examples, to provide a clear understanding of their use and significance in research studies. By categorizing variables into distinct groups based on their roles in research, their types of data, and their relationships with other variables, researchers can more effectively structure their studies and achieve more accurate conclusions.

A variable represents any characteristic, number, or quantity that can be measured or quantified. The term encompasses anything that can vary or change, ranging from simple concepts like age and height to more complex ones like satisfaction levels or economic status. Variables are essential in research as they are the foundational elements that researchers manipulate, measure, or control to gain insights into relationships, causes, and effects within their studies. They enable the framing of research questions, the formulation of hypotheses, and the interpretation of results.

Variables can be categorized based on their role in the study (such as independent and dependent variables ), the type of data they represent (quantitative or categorical), and their relationship to other variables (like confounding or control variables). Understanding what constitutes a variable and the various variable types available is a critical step in designing robust and meaningful research.

ATLAS.ti makes complex data easy to understand

Turn to our powerful data analysis tools to make the most of your research. Get started with a free trial.

Variables are crucial components in research, serving as the foundation for data collection , analysis , and interpretation . They are attributes or characteristics that can vary among subjects or over time, and understanding their types is essential for any study. Variables can be broadly classified into five main types, each with its distinct characteristics and roles within research.

This classification helps researchers in designing their studies, choosing appropriate measurement techniques, and analyzing their results accurately. The five types of variables include independent variables, dependent variables, categorical variables, continuous variables, and confounding variables. These categories not only facilitate a clearer understanding of the data but also guide the formulation of hypotheses and research methodologies.

Independent variables

Independent variables are foundational to the structure of research, serving as the factors or conditions that researchers manipulate or vary to observe their effects on dependent variables. These variables are considered "independent" because their variation does not depend on other variables within the study. Instead, they are the cause or stimulus that directly influences the outcomes being measured. For example, in an experiment to assess the effectiveness of a new teaching method on student performance, the teaching method applied (traditional vs. innovative) would be the independent variable.

The selection of an independent variable is a critical step in research design, as it directly correlates with the study's objective to determine causality or association. Researchers must clearly define and control these variables to ensure that observed changes in the dependent variable can be attributed to variations in the independent variable, thereby affirming the reliability of the results. In experimental research, the independent variable is what differentiates the control group from the experimental group, thereby setting the stage for meaningful comparison and analysis.

Dependent variables

Dependent variables are the outcomes or effects that researchers aim to explore and understand in their studies. These variables are called "dependent" because their values depend on the changes or variations of the independent variables.

Essentially, they are the responses or results that are measured to assess the impact of the independent variable's manipulation. For instance, in a study investigating the effect of exercise on weight loss, the amount of weight lost would be considered the dependent variable, as it depends on the exercise regimen (the independent variable).

The identification and measurement of the dependent variable are crucial for testing the hypothesis and drawing conclusions from the research. It allows researchers to quantify the effect of the independent variable , providing evidence for causal relationships or associations. In experimental settings, the dependent variable is what is being tested and measured across different groups or conditions, enabling researchers to assess the efficacy or impact of the independent variable's variation.

To ensure accuracy and reliability, the dependent variable must be defined clearly and measured consistently across all participants or observations. This consistency helps in reducing measurement errors and increases the validity of the research findings. By carefully analyzing the dependent variables, researchers can derive meaningful insights from their studies, contributing to the broader knowledge in their field.

Categorical variables

Categorical variables, also known as qualitative variables, represent types or categories that are used to group observations. These variables divide data into distinct groups or categories that lack a numerical value but hold significant meaning in research. Examples of categorical variables include gender (male, female, other), type of vehicle (car, truck, motorcycle), or marital status (single, married, divorced). These categories help researchers organize data into groups for comparison and analysis.

Categorical variables can be further classified into two subtypes: nominal and ordinal. Nominal variables are categories without any inherent order or ranking among them, such as blood type or ethnicity. Ordinal variables, on the other hand, imply a sort of ranking or order among the categories, like levels of satisfaction (high, medium, low) or education level (high school, bachelor's, master's, doctorate).

Understanding and identifying categorical variables is crucial in research as it influences the choice of statistical analysis methods. Since these variables represent categories without numerical significance, researchers employ specific statistical tests designed for a nominal or ordinal variable to draw meaningful conclusions. Properly classifying and analyzing categorical variables allow for the exploration of relationships between different groups within the study, shedding light on patterns and trends that might not be evident with numerical data alone.

Continuous variables

Continuous variables are quantitative variables that can take an infinite number of values within a given range. These variables are measured along a continuum and can represent very precise measurements. Examples of continuous variables include height, weight, temperature, and time. Because they can assume any value within a range, continuous variables allow for detailed analysis and a high degree of accuracy in research findings.

The ability to measure continuous variables at very fine scales makes them invaluable for many types of research, particularly in the natural and social sciences. For instance, in a study examining the effect of temperature on plant growth, temperature would be considered a continuous variable since it can vary across a wide spectrum and be measured to several decimal places.

When dealing with continuous variables, researchers often use methods incorporating a particular statistical test to accommodate a wide range of data points and the potential for infinite divisibility. This includes various forms of regression analysis, correlation, and other techniques suited for modeling and analyzing nuanced relationships between variables. The precision of continuous variables enhances the researcher's ability to detect patterns, trends, and causal relationships within the data, contributing to more robust and detailed conclusions.

Confounding variables

Confounding variables are those that can cause a false association between the independent and dependent variables, potentially leading to incorrect conclusions about the relationship being studied. These are extraneous variables that were not considered in the study design but can influence both the supposed cause and effect, creating a misleading correlation.

Identifying and controlling for a confounding variable is crucial in research to ensure the validity of the findings. This can be achieved through various methods, including randomization, stratification, and statistical control. Randomization helps to evenly distribute confounding variables across study groups, reducing their potential impact. Stratification involves analyzing the data within strata or layers that share common characteristics of the confounder. Statistical control allows researchers to adjust for the effects of confounders in the analysis phase.

Properly addressing confounding variables strengthens the credibility of research outcomes by clarifying the direct relationship between the dependent and independent variables, thus providing more accurate and reliable results.

Beyond the primary categories of variables commonly discussed in research methodology , there exists a diverse range of other variables that play significant roles in the design and analysis of studies. Below is an overview of some of these variables, highlighting their definitions and roles within research studies:

• Discrete variables : A discrete variable is a quantitative variable that represents quantitative data , such as the number of children in a family or the number of cars in a parking lot. Discrete variables can only take on specific values.
• Categorical variables : A categorical variable categorizes subjects or items into groups that do not have a natural numerical order. Categorical data includes nominal variables, like country of origin, and ordinal variables, such as education level.
• Predictor variables : Often used in statistical models, a predictor variable is used to forecast or predict the outcomes of other variables, not necessarily with a causal implication.
• Outcome variables : These variables represent the results or outcomes that researchers aim to explain or predict through their studies. An outcome variable is central to understanding the effects of predictor variables.
• Latent variables : Not directly observable, latent variables are inferred from other, directly measured variables. Examples include psychological constructs like intelligence or socioeconomic status.
• Composite variables : Created by combining multiple variables, composite variables can measure a concept more reliably or simplify the analysis. An example would be a composite happiness index derived from several survey questions .
• Preceding variables : These variables come before other variables in time or sequence, potentially influencing subsequent outcomes. A preceding variable is crucial in longitudinal studies to determine causality or sequences of events.

Master qualitative research with ATLAS.ti

Turn data into critical insights with our data analysis platform. Try out a free trial today.

• Digital Health , Health Sciences Research

Choosing the Measurements: Outcome Measurements Matter

Check our Digital Health Products

Choosing the Outcome Measurements: Planning Is the Core of Research

Conducting a medical study is a challenging process. From choosing a study design to computing a statistical analysis, medical research is complicated. On top of that, health-related studies must follow safety and ethical regulations, which adds a burden to researchers. To avoid any possible complications, planning is paramount.

Choosing the right measurements is an essential part of each study. Outcome measurements should be well-defined and accurate in order to help researchers understand the connection between variables, assess the benefits of a new intervention, and improve patients’ well-being (Peat, 2011).

Defining the Outcome and Explanatory Variables

When it comes to choosing the right measurements, explanatory and outcome variables also need to be clear and easy to measure in order to test the main hypothesis of the study. Note that the explanatory variable is a type of independent variable. While independent variables are completely autonomous and unaffected by any other factors, explanatory variables are not entirely independent (“Explanatory Variable & Response Variable: Simple Definition and Uses,” 2015).  Still, explanatory variables are vital as they can explain any possible changes and can affect the dependent variable. In fact, most phenomena are interconnected. Let’s say one wants to measure the effects of fast food and soda drinks on weight: these variables are not completely independent as food corners often offer menus that contain both options. Thus, independent and explanatory variables are two terms that are often used interchangeably. However, in precise clinical studies with multiple outcomes, all measured factors should be well-defined.

On the other hand, dependent variables, also called outcome and response variables, are the factors that are expected to change during an experiment. The outcome variable is the focus of any study, including clinical trials. For instance, experts may be interested in treatments that prolong the life of cancer patients. The type of treatment (chemotherapy, for example) will be the explanatory variable, while the survival time will be the outcome variable (“Explanatory Variable & Response Variable: Simple Definition and Uses,” 2015). Let’s not forget, though, that today’s medicine and healthcare technology focus not only on mortality rates but patients’ overall well-being and quality of life (especially in severe and chronic conditions).

Subjective Vs. Objective Outcome Measurements

Choosing the outcome measures can be a tricky task. All outcome measurements are vital for research and experts can decide on either subjective or objective outcome measurements. Both subjective and objective outcome measurements have their benefits and challenges in practice, and it’s a fact that there’s not a one-model-fits-all approach.

Subjective outcome measurements, for instance, are defined as any measurements that are open to interpretation. They can be self-administered, administered by an observer or a medical professional. One of the advantages of subjective measurements is that they are easy to administer, cost-effective, and rapid. As such, they are an effective method in clinical trials, which can assess if there’s an improvement in people’s self-reported status (Peat, 2011). Examples of subjective outcomes are questionnaires about the frequency of symptoms or illness severity. However, as the name suggests, subjective outcome measurements are based on subjective judgment, and as such, they can be prone to errors and bias. Therefore, when it comes to clinician-reported outcomes, for example, training of staff is crucial in order to avoid observer bias.

On the other hand, objective measurements gather medical information collected by standard instruments or professional equipment, which usually reduces bias. For instance, lab results and biochemistry data are examples of objective measurements (Peat, 2011). However, one of the disadvantages is that these measurements collect short-term data that changes quickly, such as blood pressure. Nevertheless, objective measurements are precise, and therefore, highly implemented in research.

Multiple Outcome Measurements

Multiple outcomes are also paramount in research, and clinical trials and experts need to consider them when choosing the outcome measurements. Since any new treatment affects various factors in one’s life, often one single measure is not enough to reflect all the physical, emotional, and social changes in patients (Tyler et al., 2011). In fact, in cases when the most important outcome is unclear, and effectiveness and efficacy must be checked across various domains, multiple measures are needed (Peat, 2011). However, when designing a study with multiple outcome measures, there must be a clear differentiation between primary and secondary outcomes in order to overcome all statistical challenges in the analysis. In fact, often researchers have a small set of measuring tools as primary outcomes and a broader one as secondary.

Note that outcomes that are significant from a medical point of view might be perceived as less important by subjects. For instance, experts may aim to reduce hospitalization rates, while patients may aim further, as returning to work.

Surrogate End-points & Clinical Trials

Note that in long-term clinical trials choosing the right outcome measurements is a delicate process. In fact, the primary outcome variable is called a surrogate end-point or an alternative short-term outcome. Surrogate end-points are defined as biomarkers “intended to substitute for a clinical endpoint” (Aronson, 2005). Surrogate end-points facilitate research as they are easy to implement and more cost-effective. On top of that, there are some ethical issues that allow the use of surrogate end-points only. For instance, in laboratory settings and physiological markers, blood pressure indicators may be used as a surrogate for stroke. Still, when the primary outcome is mortality, end-points cannot substitute the true end-points in the long term and more research is needed to confirm findings and benefits of treatment.

In general, the clinical trial is a complicated process, which is often marked by low recruitment rates, financial demands, and ethical regulations. Research is needed as new drugs, and alternative treatments can improve people’s well-being and save lives.

Outcome Measurements: Qualities to Consider

No matter what outcome measurements researchers choose, there are some essential qualities that all measures need to cover:

All measurements need to reveal good validity. Validity can be explained as the degree to which a measurement is valid and strong in what it claims to measure. For instance, intelligence tests need to measure intelligence, not memory, in order to be valid.

• Face validity is one of the essential types of validity . As the name suggests, face validity describes if a measurement is at face value and assesses what appears to be measured. In clinical settings, face validity shows if the outcome measures identify important symptoms and changes.
• Content validity or logical/rational validity shows if a measurement manages to measure every facet of a theoretical construct. In medicine, content validity guarantees that measurement is relevant to the study and the illness in general.
• Criterion validity reveals how well a measurement can predict a health-related outcome and correlates with other research measures.
• Construct validity can be defined as the extent to which a test measures the construct it’s supposed to test.

Reliability & Repeatability:

Reliability or the consistency of measurement is also crucial. The repeatability of measurement is essential because good test-retest reliability can avoid variability. In other words, the same test given to the same people in a short period should show the same results. The between-observer agreement should also be sought. Note that between-observer agreement or inter-rater reliability reveals the consensus between different raters.

Errors should be diminished by good outcome measurements. There are three types of systematic errors (Peat, 2011):

• Subject errors are systematic errors related to subjects’ bias.
• Observer errors are due to differences in observers and the administering and interpreting of a test.
• Instrument errors are caused by the instrument itself. Thus, tools should be accurate and calibrated according to a standard (certain temperature, for example).

Responsiveness:

Responsiveness is another crucial quality (Tarrant et al., 2014). It measures the efficacy and effectiveness of an intervention and the extent to which the service quality meets clients’ needs. Measurements should be responsive to within-subject changes. Some tools cannot detect small changes, and therefore, they cannot be used as primary outcomes. For example, measurements, such as a 5-point scale in which the symptom frequency is categorized as ‘constant,’ ‘frequent,’ ‘occasional,’ ‘rare’ or ‘never,’ are not responsive; such tools cannot detect subtle changes in symptoms. On top of that, experts should consider not only physiological changes but the quality of life.

Sample Size:

The statistical power of a test is another factor to consider when choosing outcome measurements (Peat, 2011). In general, the sample size needs to be adequate in order to show clinical and statistical differences between groups. Often, studies focus only on primary outcomes, but in practice, a broad range of outcomes should be implemented in research.

Sample size and types of variables go together. Here we should mention that categorical values, for instance, need bigger samples. A categorical or nominal variable has two or more categories (without any order). Gender is a categorical variable with two categories (male and female), and there is no intrinsic ordering of the categories. Ordinal values, on the other hand, are similar but there’s order in the categories. Note that the interval of the categories and their order is inconsistent, though. Economic status – with low, medium, and high categories – is an example of ordinal values. On the other hand, interval values have also categories placed in order but with equal spacing in between. Income of $5,000, $10,000, and $15,000 is a good example, as the size of that interval is the same ($5,000). Last but not the least, continuously distributed measurements, such as blood pressure, are vital (“What is the difference between categorical, ordinal and interval variables?” 2017).

To sum up, choosing the outcome measurements is a serious step that lies in the path to success of any health-related study. Because outcome measurements matter!

• Aronson, J. (2005). Biomarkers and surrogate endpoints. British Journal of Clinical Pharmacology, 59(5), p.491-494.
• Explanatory Variable & Response Variable: Simple Definition and Uses (2015, February 16). Retrieved from http://www.statisticshowto.com/explanatory-variable/
• Peat, J. (2011). Choosing the Measurements. Health Science Research, SAGE Publications, Ltd.
• Tarrant, C., Angell, E., Baker, R., Boulton, M., Freeman, G., Wilkie, P., Kackson, P., Wobi, F., & Ketley, D. (2014). Responsiveness of primary care services: development of a patient-report measure – qualitative study and initial quantitative pilot testing. Health Services and Delivery Research, No. 2, 46.
• Tyler, K., Normand, S., & Horton, N. (2011). The Use and Abuse of Multiple Outcomes in Randomized Controlled Depression Trials. Contemporary Clinical Trials, 32(2), p. 299-304.
• What is the difference between categorical, ordinal, and interval variables? (2017). Retrieved from https://stats.idre.ucla.edu/other/mult-pkg/whatstat/what-is-the-difference-between-categorical-ordinal-and-interval-variables/

We’ve collected the items for you to purchase for your convenience.

Get the entire package for up to 50% discount with our Replication program.

Our Location

Conduct science.

• Become a Partner
• Social Media
• Career /Academia
• Privacy Policy
• Shipping & Returns
• Request a quote

Customer service

• Account details
• Lost password

DISCLAIMER: ConductScience and affiliate products are NOT designed for human consumption, testing, or clinical utilization. They are designed for pre-clinical utilization only. Customers purchasing apparatus for the purposes of scientific research or veterinary care affirm adherence to applicable regulatory bodies for the country in which their research or care is conducted.

In order to continue enjoying our site, we ask that you confirm your identity as a human. Thank you very much for your cooperation.

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• ROBVALU: a tool for...

ROBVALU: a tool for assessing risk of bias in studies about people’s values, utilities, or importance of health outcomes

• Related content
• Peer review
• Samer G Karam , doctoral student 1 2 ,
• Yuan Zhang , assistant clinical professor 1 2 ,
• Hector Pardo-Hernandez , researcher 3 4 ,
• Uwe Siebert , professor 5 6 7 ,
• Laura Koopman , senior adviser 8 ,
• Jane Noyes , professor 9 ,
• Jean-Eric Tarride , professor 1 10 11 ,
• Adrienne L Stevens , manager 12 ,
• Vivian Welch , senior investigator 13 ,
• Zuleika Saz-Parkinson , project adviser 14 ,
• Brendalynn Ens , director (retired) 15 ,
• Tahira Devji , medical student 16 ,
• Feng Xie , professor 1 10 ,
• Glen Hazlewood , associate professor 17 18 ,
• Lawrence Mbuagbaw , associate professor 1 19 20 21 22 23 ,
• Pablo Alonso-Coello , senior researcher 3 4 24 ,
• Jan L Brozek , associate professor 1 2 ,
• 1 Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada
• 2 Michael G DeGroote Cochrane Canada and McMaster GRADE Centres, McMaster University, Hamilton, ON, Canada
• 3 Iberoamerican Cochrane Centre, Sant Antoni Maria Claret, Barcelona, Spain
• 4 Centro de Investigación Biomédica en Red de Epidemiología y Salud Pública (CIBERESP), Madrid, Spain
• 5 Department of Public Health, Health Services Research and Health Technology Assessment, Institute of Public Health, Medical Decision Making and Health Technology Assessment, UMIT TIROL-University for Health Sciences and Technology, Hall in Tirol, Austria
• 6 Center for Health Decision Science and Departments of Epidemiology and Health Policy and Management, Harvard T H Chan School of Public Health, Boston, MA, USA
• 7 Institute for Technology Assessment and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
• 8 Department of Specialist Medical Care, National Health Care Institute, Diemen, Netherlands
• 9 School of Medical and Health Sciences, Bangor University, Wales, UK
• 10 Centre for Health Economics and Policy Analysis, McMaster University Faculty of Health Sciences, Hamilton, ON, Canada
• 11 Programs for Assessment of Technologies in Health, St Joseph’s Healthcare Hamilton, Hamilton, ON, Canada
• 12 Centre for Immunisation Programmes, Public Health Agency of Canada, ON, Canada
• 13 Bruyère Research Institute and, School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada
• 14 European Commission, Joint Research Centre, Ispra, Italy
• 15 Implementation Support and Knowledge Mobilisation, Canadian Agency for Drugs and Technologies in Health, Ottawa, ON, Canada
• 16 Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
• 17 Department of Medicine, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
• 18 Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
• 19 Department of Anaesthesia, McMaster University, Hamilton, ON, Canada
• 20 Department of Paediatrics, McMaster University, Hamilton, ON, Canada
• 21 Biostatistics Unit, Father Sean O’Sullivan Research Centre, St Joseph’s Healthcare, Hamilton, ON, Canada
• 22 Centre for Development of Best Practices in Health, Yaoundé Central Hospital, Yaoundé, Cameroon
• 23 Division of Epidemiology and Biostatistics, Department of Global Health, Stellenbosch University, Cape Town, South Africa
• 24 Institut de Recerca Sant Pau (IR SANT PAU), Sant Quintí, Barcelona, Spain
• 25 Clinical Epidemiology and Research Centre (CERC), Humanitas University and Humanitas Research Hospital, Via Rita Levi Montalcini 4, 20090 Pieve Emanuele, Milan, Italy
• Correspondence to: H J Schünemann schuneh{at}mcmaster.ca
• Accepted 9 April 2024

People’s values are an important driver in healthcare decision making. The certainty of an intervention’s effect on benefits and harms relies on two factors: the certainty in the measured effect on an outcome in terms of risk difference and the certainty in its value, also known as utility or importance. The GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) working group has proposed a set of questions to assess the risk of bias in a body of evidence from studies investigating how people value outcomes. However, these questions do not address risk of bias in individual studies that, similar to risk-of-bias tools for other research studies, is required to evaluate such evidence. Thus, the Risk of Bias in studies of Values and Utilities (ROBVALU) tool was developed. ROBVALU has good psychometric properties and will be useful when assessing individual studies in measuring values, utilities, or the importance of outcomes. As such, ROBVALU can be used to assess risk of bias in studies included in systematic reviews and health guidelines. It also can support health research assessments, where the risk of bias of input variables determines the certainty in model outputs. These assessments include, for example, decision analysis and cost utility or cost effectiveness analysis for health technology assessment, health policy, and reimbursement decision making.

Healthcare decision making relies on evidence on the relative effectiveness, safety, and cost effectiveness of an intervention evaluated in appropriate studies. 1 2 Choosing between different interventions (such as preventive, diagnostic, or treatment strategies) depends on the importance or value that people place on specific health states or health outcomes. 2 Values have a major role at different levels of decision making, from the individual level to the healthcare system level. In this context, people’s values reflect the importance they place on outcomes of interest that result from decisions about using an intervention—for example, taking a certain test or starting a new treatment regimen. 2 We use the term “people” when talking about value because the term is inclusive to patients, healthcare providers, policy makers, and the general public.

Utility instruments are widely used to elicit the absolute value of a health outcome, and provide an index measure anchored on a scale with 1 reflecting perfect health and 0 reflecting being dead. 3 4 Various methods are used to establish values, including direct measures of utility, indirect measurements of utility, or qualitative research. 2 5 The visual analogue scale (VAS) is one of the simplest measures to elicit these values. People are asked to rate a health state on a VAS that is then converted to a utility value. 6 7 While the scale directly measures the importance of an outcome, concerns exist about how accurate and valid it might be. 2 Other direct measures such as the standard gamble and time trade-off require people to choose between their current health state and a treatment option that could result in perfect health or in immediate death. 4 8 Discrete choice experiments ask people to choose between two or more treatment options where the choices differ in terms of their attributes, that are defined by the investigators. 9 The relative importance of each attribute is then inferred by analysing the responses, assuming that patients choose the option with the highest value. 9 Indirect methods of measuring utility values include validated, health related, quality-of-life instruments, such as the EQ-5D and the Health Utilities Index. 10 The EQ-5D requires respondents to answer questions across five domains that are converted to a utility value using validated scoring systems. 11 12

Summary points

Assessing the risk of bias in individual studies is an essential step to determine overall certainty of evidence in a systematic review or health technology assessment and for guideline development

The Risk of Bias in Values and Utilities (ROBVALU) tool assesses risk of bias in quantitative studies of people’s values, utilities, or importance of outcomes

A sequential mixed methods approach was used to develop ROBVALU, initially based on signalling questions and subdomains developed by the GRADE working group to assess risk of bias; a modified Delphi approach was used for final refinement of the tool

ROBVALU covers four separate subdomains through which bias might be introduced; individual subdomain judgments inform the overall risk of bias of studies

ROBVALU has demonstrated high validity and reliability

General application of utility values in research

These utility values allow researchers to weigh the benefits and harms of an option and, thus, they also are important in health economics and health technology assessments. 3 13 For instance, in decision analysis, they are required to calculate quality adjusted life years. Confidence in studies that report on values needs to be ascertained for decision making in guideline recommendations, health technology assessments, or coverage decision. 14 For example, in a systematic review on people with chronic obstructive pulmonary disease, we found moderate certainty that patients value adverse events as important, but on average valued them as less important than symptom relief. 15 We also found moderate certainty that exacerbation and hospital admission owing to exacerbation are the outcomes that patients with chronic obstructive pulmonary disease rate as most important. In another example, a systematic review on patients’ values on venous thromboembolism, we found that people with cancer placed more importance on a reduction in new or recurrent venous thromboembolism than on a decrease in major or minor bleeding events. 16

The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) Evidence to Decision frameworks is a widely used approach in guidelines, health technology assessment, and other decisions. The frameworks require judgments about the certainty in how much people value the main outcomes: “Is there important uncertainty about . . . how much people value the main outcomes?” 17 18 A key determinant of certainty is internal validity—that is, how well individual studies were designed and conducted (ie, internal validity, which GRADE and Cochrane label as the risk-of-bias (ROB) domain).

Risk of bias

Similar to other study designs, threats to internal validity arising from the study design, conduct, analysis, and reporting of the study introduce ROB in research on utility values. 2 Poor study quality could result in indirectness which encompasses applicability and external validity, often as a result of PICO (patient/population, intervention, comparison, and outcomes) elements. Another quality issue is low sample size or no sample size calculation, which could result in imprecision. ROB assessment tools are developed to assess biases that result in threats of internal validity and would not measure indirectness and precision. Quality assessment tools and reporting checklists often include all factors of a study’s qualities and safeguards, but these tools differ from a ROB assessment tool that aims to present a ROB judgment for a study. A key factor that might introduce bias in values studies is the instrument used to measure utilities of the people in the study. Bias means that a value people place on an outcome in a research study (eg, a value of 0.5 for stroke) would be systematically different from the true value that people would place on that outcome. For example, the true unbiased value might be 0.3 and, thus, use of biased estimates would provide inaccurate answers in the modelling and health decision making context.

ROB assessment tools exist for many study designs, including the Cochrane Risk of Bias 2 (RoB 2) for randomised trials, 19 ROBINS-I for non-randomised studies of the effects of interventions, 20 and ROBINS-E for studies about exposures. 21 22 Critical appraisal tools to assess the quality of a study are also study design specific, such as the Newcastle-Ottawa scale and the Joanna Briggs Institute’s critical appraisal tool for cross sectional studies. 23 24 These tools are regularly used by researchers to assess the quality of individual studies or to assess ROB, but they were not developed for studies on utility values. These checklists invariably include questions specific to the study design, which would not always be appropriate to answer in studies about people’s values (eg, “Were there deviations from the intended intervention that arose because of the trial context?” or “Was the exposure measured in a valid and reliable way?”). 19 21 22 For studies on utility values, a major concern that is not adequately addressed by any commonly used ROB tool is the method used to elicit people’s values. The measurement instrument needs to be valid and reliable, be used appropriately, use valid health outcomes, and explore proper understanding of the instrument. No validated tool is available for the nuanced assessment of ROB in individual studies measuring utility values. 9 20 25 26

To properly implement evidence based decision making and formulate evidence based recommendations in clinical or public health guidelines, evaluation of ROB is crucial in studies of values, utilities, or importance of outcomes. However, owing to the absence of specialised and validated tools to assess ROB, this evaluation is rarely done. Thus, our goal was to develop, validate, and describe a pragmatic tool for studies measuring the value people place on health outcomes with appropriate guidance to apply it correctly.

Development of the ROBVALU tool and guidance

We used a sequential, mixed methods approach to develop ROBVALU and related guidance document (supplement S1), 27 starting with a qualitative approach and followed by a quantitative phase to assess the psychometric properties of the tool ( fig 1 ). In the qualitative phase, we first considered the ROB signalling questions (appendix table A1) and subdomains that we had developed for GRADE guidance to assess ROB about values across studies in a body of evidence. 2 For that GRADE guidance, we iteratively developed the subdomains and signalling questions starting with a list of 23 items identified as part of a systematic survey project. 26 The core research group reviewed the 23 items to identify any missing item that might be relevant for the single study ROBVALU tool. After thorough group discussions, a decision was made not to add any new items or subdomains to avoid complexity, thereby improving applicability, feasibility, and adoption of the tool.

Tool development process for the Risk of Bias in Values and Utilities (ROBVALU) tool. GRADE=Grading of Recommendations, Assessment, Development, and Evaluation

• Download figure
• Open in new tab
• Download powerpoint

We first structured a preliminary version of the tool and added simple considerations to help answer the signalling questions. These signalling questions were categorised into four subdomains: selection of participants into the study, completeness of data, measurement instrument, and data analysis. We used a 4 point, Likert-type scale (ie, yes, probably yes, probably no, no) to judge the individual items, to avoid a neutral option of a 5 point Likert scale when studies lack sufficient information to make a proper judgment. In each subdomain, the tool asked how important and how serious the ROB issue is. The core research group iteratively revised the tool and accompanying guidance document. An advisory group of experts provided feedback and suggested appropriate changes to establish face and content validity (supplement S2).

Participant testing

We used purposeful sampling to recruit 15 participants with experience in critical appraisal, systematic reviews, or guidelines for user testing and semi-structured interviews (supplement S3). The participants had a broad level of expertise, from masters level students to senior researchers with experience in health research ranging from six months to 30 years (appendix table A2). All users received the ROBVALU tool and the accompanying guidance document (supplement S1). We instructed the participants to complete three to four assessments and every sample study was assessed by four users independently, 11 studies in total were assessed (appendix table A3). Based on feedback received in the semi-structured interview after user testing, we iteratively revised and improved the guidance document throughout the project with a focus on the wordings, spelling, and grammatical structure of the guidance document. The ROBVALU tool demonstrated good psychometric properties with an overall intraclass correlation coefficient of 0.87 and the four subdomains showed good to excellent reliability ranging from 0.80 to 0.91 ( table 1 and supplement S4). We also calculated the inter-rater reliability of the global ROB judgment using the ROBVALU tool with Kendall’s W, which showed substantial agreement of 0.62 (supplement S4). We invited four expert participants in the field to provide a global judgment for ROB without using the ROBVALU, with each expert rating three to four studies. When we added expert participant responses of the global ROB judgment, the Kendall’s W dropped to 0.45, showing moderate agreement (supplement S4). However, only four global judgment responses were more than one level of seriousness higher or lower than the expert participant judgment (appendix table A4).

Reliability of the Risk of Bias in Values and Utilities (ROBVALU) tool. CI=confidence interval

• View inline

Modified Delphi process

Finally, following our protocol, we used purposeful sampling to invite 20 experts in values, utilities, health technology assessment, and health decision science to participate in a modified Delphi process for final refinement of the tool (supplement S5, fig S8). 28 29 30 We used our extensive network of global colleagues working in the field of study to identify and invite the expert panel. Ten voting members accepted the invite to participate in the Delphi panel, and four members of the working group participated as non-voting members. We shared the ROBVALU tool draft, guidance document, and results of our participant testing with the panel members.

The first round of the Delphi process involved an anonymous survey to determine the signalling questions to be included. The second round took place via recorded video conferences with the aim of identifying common themes and reaching consensus on simplifying and harmonising language across the tool. The third and final round of the Delphi process included an anonymous survey for final consensus on the wording of the signalling questions and the proposed methods for providing a global ROB judgment. We used Google forms to prepare the surveys; the first survey used a 7 point Likert scale (ie, strongly agree, agree, somewhat agree, neutral, somewhat agree, disagree, and strongly disagree) to rate each item, with 70% agreement set as the cut-off threshold to retain or remove a signalling question. The final survey used a 3 point scale (ie, agree, neutral, and disagree) with a 70% agreement set as the cut-off threshold to retain the signalling question.

We had a 100% response rate in the first round of the Delphi process, with 80-100% consensus to retain all signalling questions. We also collected feedback from open ended questions for suggested edits for the signalling questions (supplement S6). In the second round of the Delphi process, we presented the ROBVALU tool, psychometric properties, exploratory factor analysis, and results of the first round of the Delphi to the panel members. After deliberating on the tool’s properties, agreement was reached to edit some signalling questions to simplify the language or to harmonise the language across the tool, which resulted in minor changes only. We also discussed how to make a final judgment for ROB for a study.

We had a 100% response rate in the third and final Delphi round, with 80-100% consensus on the tool’s signalling questions, including those with minor adjustments to the wording. We also established a consensus of >70% that the overall ROB judgment should match the most severe ROB judgment on an item, unless appraisers can provide justifications to rate the overall ROB lower (eg, many concerns on many items) or higher (eg, concern seems not to have an important influence on overall ROB). For example, if multiple subdomains were rated as very serious, the final judgment could be rated as extremely serious (supplement S7).

Risk-of-bias subdomains

ROBVALU includes seven key signalling questions across four subdomains: selection of participants into the study, completeness of data, measurement instrument, and data analysis ( table 2 ).

Subdomains and considerations in the Risk of Bias in Values and Utilities (ROBVALU) tool

Selection of participants into the study

Precise research questions include a clear definition of the target population. The study population of any empirical study must be representative for this target population, and is therefore, a critical component because bias in the selection will lead to biased estimates of the values people place on outcomes in the target population. 2 When assessing selection bias, users should consider the study’s sampling strategy, in particular if the achieved sample population deviates from the intended sample population, 2 because this might lead to biased estimates for the study’s population of interest owing to threats to internal validity. If the achieved sample population does not deviate from the intended sample population but differs from the population researchers intend to extrapolate the results to, this difference will result in a lack of generalisability. We refer to this lack of generalisability as indirectness, which encompasses applicability and external validity. The ROBVALU tool is not intended to deal with indirectness, a different domain in assessing the certainty of a body of evidence according to GRADE, but we are developing a tool that is specific to indirectness separately.

Completeness of data

When judging completeness of data, reviewers need to consider the response rate of the study population, the attrition rate if follow-up was involved, and the differential responders compared with non-responders. 2 High response rates and low proportion of loss to follow-up are clearly preferable, and a high proportion of non-response or dropout rates could be problematic. 2 Participants providing responses could plausibly differ from those who do not, and researchers should consider that results coming only from those participants who responded or completed follow-up might be misleading. 2

Measurement instrument

Reliable and valid instruments should be used to measure the relative importance of outcomes in values, preferences, and utility studies. 2 Using unreliable or poorly validated instruments can result in biased measurements of the outcome. Similarly, utility values for specific health states based on instruments not sufficiently validated that are used as input parameters for decision analytical models can result in biased estimates, such as quality adjusted life years derived from state transition models. 31 32 Researchers conducting primary empirical studies should provide information regarding the measurement properties of their chosen instrument. 2

Researchers should also demonstrate that the instrument has been used correctly and in a consistent manner across all participants in a study. For example, if the standard gamble is to be administered by an interviewer, but a subset of participants used self-administration, this could result in biased utility estimates that could be due to systematic differences between the two groups. In addition, an optimal representation of the outcome or health state should be presented or described in a way that accurately reflects the attribute the researchers intended to measure. This information could include a detailed explanation of how the outcome defines the experience, the probability of the outcome, durations, and possible consequences. Finally, researchers should evaluate whether participants had a proper understanding of the instrument to complete the tasks.

Data analysis

Studies should explore heterogeneity in values when appropriate and present results for the different subgroups. The data analysis plan and exploration of heterogeneity should be outlined a priori before collection of data. A causal framework that helps delineate health state and outcome interactions with possible confounding factors will help make assumptions explicit. If heterogeneity is found, the evaluator needs to consider whether the adjustment, stratification, or model selection used in the study reporting on values was appropriate. 2 Adjusting for important confounding factors (such as age if it is associated with the intervention and influences the estimated values) or reporting values in a stratified manner reduces biased estimates of the value placed on an outcome. In addition, self-inflicted biases, including selection bias or immortal time bias should be controlled for appropriately using modern causal inference methods (eg, target trial emulation or g methods for time varying confounding). 33

ROBVALU tool application

The assessment of ROB in studies evaluating the value people place on outcomes follows seven steps:

Specify the research or review question.

Specify the outcome being assessed.

Identify the sampling frame, the response rate and/or attrition rate, the measurement instrument used, and the data analysis plan.

Answer the signalling questions of the four subdomains.

Make a judgment if the four subdomains have important ROB concerns.

Formulate a ROB judgment for the four subdomains.

Formulate an overall ROB judgment for the study outcome being assessed.

The ROBVALU tool (supplement S8) provides users with space to record vital information of the study being assessed, and signalling questions to all four subdomains that must be answered. We validated a 4 point Likert-type scale (yes, probably yes, probably no, no) to respond to the individual signalling questions (items). When rating individual signalling questions, we suggest following the flowchart in figure 2 for consistent answers between raters. In each subdomain, the tool asks to specify how important the ROB issue is on a 4 point Likert-type scale (yes, probably yes, probably no, no), and how serious the overall ROB issue is on a 4 point Likert-type scale (not serious, serious, very serious, extremely serious). Responses to the signalling questions should provide the basis for the subdomain level judgment, of how important and how serious the ROB issues are in the study. Raters should provide a rationale for the response as free text, to justify their judgments. We suggest that the final judgment for each subdomain inversely correlates with the signalling question judgment. For example, in the measurement instrument subdomain, if the answer to “Was the instrument administered in the intended way?” was “No,” then the answer to “Are there important risk of bias issues concerning the measurement instruments?” should be “Yes.” If raters believe that the lowest signalling question judgment does not reflect the overall subdomain judgment, they might choose not to deem the results of the study at ROB for that subdomain, but they are asked to provide explanations for why they would not do this.

Rating individual signalling questions in the Risk of Bias in Values and Utilities (ROBVALU) tool

The global ROB judgment for a study corresponds to the lowest subdomain judgment ( table 3 ), because any domain level bias will lower our confidence in the study results. If users do not believe that the lowest subdomain judgment reflects the global ROB judgment, they should provide a justification. For example, if a study has a low response rate resulting in very serious ROB domain judgment and the study results are comparable to better quality studies, a reviewer might consider that the subdomain judgment does not reflect the global ROB judgment. Box 1 presents an illustrative example of a completed assessment (supplement S9).

Response options for judgments on risk of bias at an overall study level, according to the Risk of Bias in Values and Utilities (ROBVALU) tool

Example application of the Risk of Bias in Values and Utilities (ROBVALU) tool to assess risk of bias in values assigned to exacerbation of chronic obstructive pulmonary disease 34

In a study of 65 men and women with chronic obstructive pulmonary disease, researchers assessed the utility value that participants placed on an exacerbation, at seven study sites in the US when they visited an outpatient clinic within 48 hours of symptom onset. 34 Eligible participants were at least 40 years old and were current or former smokers with a history of at least 10 pack years. Of 65 participants, 59 completed the study, three were lost to follow-up, and three were ineligible. Utility values were measured using the EQ-5D.

An assessment using the ROBVALU tool revealed the following (supplement 9):

Selection of participants into the study would likely lead to risk of bias.

Exacerbations that required hospital admission were considered severe and were excluded from this study and might importantly bias the estimates. Thus, the population was deemed to be probably not representative of the intended population.

Completeness of data was present:

Only three patients were lost to follow-up, which did not cause risk of bias.

Measurement instrument caused some concern about risk of bias:

It was not clear whether the instrument was used in a valid and reliable manner, but it was applied in the intended way using a valid representation of the outcome. Patients also appeared to show an understanding of the instrument that was used and did not encounter difficulties, but this was not reported.

Data analysis did not cause concern for risk of bias:

Adjustment, stratification, and model selection was appropriate based on a plan created a priori.

ROBVALU assessment

Overall risk of bias was deemed serious because of issues related to the selection of participants into the study and the way the measurement instrument was used.

We have developed and validated the ROBVALU tool, a new instrument to assess ROB in studies measuring the value, utility, or relative importance that people place on health outcome. We followed a sequential mixed methods approach, by first adapting the signalling questions from the GRADE guidance for judging ROB across studies. ROBVALU differs from existing GRADE guidance by specifically assessing ROB in individual studies as opposed to across studies. 2 We iteratively revised the tool with our core group and an advisory group. The final draft tool contains 15 items in four subdomains: selection of participants, completeness of data, measurement instrument, and data analysis. We conducted a validation exercise with 15 participants that showed good reliability. Additional refinement using a modified Delphi process established construct validity on the final content of the tool.

Strengths and limitations

Assessing ROB is an essential step to assess the overall certainty of the evidence in a systematic review or health technology assessment and to develop a guideline. This assessment has often relied on adapting ROB tools not specifically designed for this type of research. 26 However, the lack of validation could lead to unreliable certainty of the evidence assessments, both for single studies and for a body of evidence. By using ROBVALU, evaluators can incorporate the ROB assessment into their meta-analysis, such as performing a sensitivity analysis to evaluate how studies with higher ROB might affect the study’s conclusion or primary outcomes. An advantage of the ROBVALU tool is the use of standardised GRADE terminology and judgments to facilitate assessment when establishing the certainty of the evidence. The ROBVALU tool can also be used to assess ROB in all elicitation studies of values, utilities, and importance of outcomes that use discrete choice, ranking, indifference, and rating methods. 35 Finally, the tool can be used in individual studies that use indirect methods to elicit people’s preferences, such as quality of life and EQ-5D scores.

This study and the derived tool also has several limitations. The new tool focuses on assessing values quantitively. For any given intervention, there is usually qualitative literature exploring what patients want to achieve and what they value (or not) from interventions; this information could be important for decision making. While some of the signalling questions might be used for qualitative studies, other signalling questions will not apply. Further exploration with qualitative studies should be performed to assess how ROBVALU can be adapted for that particular use, or whether a different tool is required. Furthermore, an exploratory factor analysis showed that one item in the tool had relatively poor fit (Was a valid representation of the outcome (health state) used?), but this poor fit could be due to the relatively small sample size. However, we retained this item because of feedback from the Delphi panel, who deemed it important. External validation of ROBVALU’s reliability by different users and on different studies will help refine the guidance and the tool.

Future implications

ROBVALU allows researchers to appraise individual studies reporting utilities, values, or the importance of outcomes for risk of bias. For example, in health technology assessments, the certainty of input variables from an individual study determines the certainty of outputs from decision analytical models (eg, cost utility and cost effectiveness analyses). 32 36 ROBVALU should also help with evaluating ROB as part of a systematic review, health technology assessment, or formal health guideline, to develop recommendations and make judgments across the overall body of this type of evidence (eg, assessing overall certainty of the evidence when following the GRADE approach).

Ethics statements

Ethical approval.

This international study was designed and coordinated at McMaster University after approval by the Hamilton Integrated Research Ethics Board (project ID 5634), and interviews and meetings were conducted in person or over video conference. All participants provided informed consent.

Contributors: The authors are epidemiologists, statisticians, systematic reviewers, and health services researchers, many of whom are involved with methods research and GRADE. Development of ROBVALU was informed by GRADE guideline 19, previously published tools for assessing risk of bias in intervention studies, systematic reviews of available tools to assess risk of bias in values and preferences, and the authors’ experience of developing similar tools to assess risk of bias. All authors contributed to development of the ROBVALU tool and to writing associated guidance. SGK, YZ, JLB, and HJS designed the study and formed the core group. YZ, JLB, and HJS conceived of the project. HJS oversaw the project and is guarantor. SGK, YZ, TD, JLB, and HJS drafted the ROBVALU tool. JN, PAC, FX, and US formed the advisory group. SGK led working groups and conducted the semi-structured interviews. SGK and LM analysed the data. HP-H, GH, YZ, and PAC assessed studies. PAC, FX, BE, ZSP, VW, ALS, J-ET, JN, LK, and US participated in the Delphi process as voting members, and HJS, YZ, SGK, and JLB were non-voting members. SGK and HJS drafted the manuscript. YZ, JLB, and HJS obtained funding for the study. All authors reviewed and commented on drafts of the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding: The study was funded by the Canadian Institutes of Health Research (grant 401310 to HJS and JLB). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: support from the Canadian Institutes of Health Research for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Provenance and peer review: Not commissioned; externally peer reviewed.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ .

• ↵ Boyd CM, Singh S, Varadhan R, et al. Methods for benefit and harm assessment in systematic reviews. 2012. https://www.ncbi.nlm.nih.gov/books/NBK115750/pdf/Bookshelf_NBK115750.pdf
• Alonso-Coello P ,
• Guyatt GH ,
• Pieterse AH ,
• Stiggelbout AM
• McDonough CM ,
• Tosteson AN
• Torrance GW ,
• Rashidi AA ,
• Bleichrodt H ,
• Johannesson M
• Bridges JF ,
• Hauber AB ,
• Marshall D ,
• Horsman J ,
• Furlong W ,
• Slaughter KB ,
• Bambhroliya AB ,
• Schünemann HJ ,
• Piggott T ,
• Morgan RL ,
• Etxeandia-Ikobaltzeta I ,
• Brundisini F ,
• GRADE Working Group
• Kallenbach M ,
• Meerpohl J ,
• Sterne JAC ,
• Savović J ,
• Sterne JA ,
• Hernán MA ,
• Reeves BC ,
• Thayer KA ,
• Santesso N ,
• Higgins JPT ,
• Rooney AA ,
• ↵ Wells GA, Shea B, O’Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. 2000.
• ↵ Joanna Briggs Institute. The Joanna Briggs Institute critical appraisal tools for use in JBI systematic reviews checklist for analytical cross sectional studies. 2017.
• Mokkink LB ,
• Terwee CB ,
• Patrick DL ,
• Yepes-Nuñez JJ ,
• Creswell JW ,
• ↵ Helmer -Hirschberg O. Analysis of the future: The Delphi method. Rand, 1967.
• Murphy MK ,
• Lamping DL ,
• Siebert U ,
• Bayoumi AM ,
• ISPOR-SMDM Modeling Good Research Practices Task Force
• Arvandi M ,
• Goossens LM ,
• Nivens MC ,
• Rutten-van Mölken MP
• Soekhai V ,
• Whichello C ,
• Levitan B ,
• Briggs AH ,

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Smith PG, Morrow RH, Ross DA, editors. Field Trials of Health Interventions: A Toolbox. 3rd edition. Oxford (UK): OUP Oxford; 2015 Jun 1.

Field Trials of Health Interventions: A Toolbox. 3rd edition.

Chapter 12 outcome measures and case definition, 1. introduction to outcome measures and case definition.

Field trials of health interventions are designed to assess the impact of one or more interventions on the incidence, duration, or severity of specified diseases, or on intermediate variables or risk factors considered to be closely related to these measures of disease (for example, hygiene behaviours for diarrhoeal diseases, reduction in density of parasite vector, reduction of indoor air pollutants for pneumonia, or reduction of salt intake for hypertension). The measures chosen to assess the impact of the interventions are called the outcome measures in the trial (or the trial endpoints). Such measures should be defined at the time the trial is designed and should be specified in detail in the study protocol. The outcomes should be compared between those in the different intervention groups and should be measured in a consistent way during the course of the trial in the different groups. Clear definitions are also necessary, so that the measures can be replicated in other trials and meaningful comparisons made between trials. Failure to pay sufficient attention to the precise definition of the primary outcome measures at the start of a trial may lead to confusion in interpreting the results or can even invalidate them.

As discussed in Chapter 4 , Section 5 , several different outcome measures may be employed in a trial. It is important to decide which is of most interest (primary outcome), as this has major design implications, particularly in terms of the study size and duration. Trials may have other outcomes (secondary or tertiary) that may be important to measure, although they will generally not determine the size of the trial. In Table 12.1 , there are some examples of primary and secondary outcomes for trials of different interventions.

Examples of primary and secondary outcomes for trials of different interventions.

In this chapter, different types of outcome measures are reviewed in Section 2 , and factors influencing the selection of these are discussed in Section 3 . The importance of standardizing measurements between different observers is stressed in Section 4.1 , and there is a discussion of how the results of a trial may be influenced by poor sensitivity or specificity in the outcome measures in Section 4.2 . Finally, ways of avoiding bias and maintaining quality control (QC) in case ascertainment methods are reviewed in Sections 4.3 and Sections 4.5 .

2. Types of outcome measures

2.1. primary, secondary, tertiary, 2.1.1. primary outcomes.

Primary outcomes are the most important outcomes of the study, the ones that determine its design and the study size. They represent the main reason the trial is being conducted. Normally, a trial has only one primary outcome, so, for each main question in the development of a new drug, vaccine, or intervention, one specific trial is usually conducted. However, more than one primary outcome may be selected in some trials, provided the design and sample size allow it and if measuring them in the study does not substantially add to the cost or complicate the design or conduct of the trial. For example, Phase I or II clinical trials usually have several primary outcomes (such as the safety of a new drug or vaccine, evaluated through a series of clinical outcomes, as well as the immunogenicity of the vaccine or pharmacodynamics of the drug). Phase III or IV trials have fewer primary outcomes, and often only one. Primary outcomes need careful definition prior to the start of the trial (indicator, instrument to be used, measurement to be taken, values which will be considered as a positive or negative result, which laboratory will be used, etc.); these should be agreed upon among investigators, sponsors, and any regulatory agencies overseeing the trial.

2.1.2. Secondary and tertiary outcomes

Trials often have additional important outcomes, but these are not usually used to determine the trial design and sample size. They are included as secondary or tertiary outcomes to be measured in the trial. These outcomes may not be statistically conclusive, since the trial may not have been designed with the power to evaluate them, but they can be very useful to generate further hypotheses and guide future trials. Because of their importance in justifying future studies, these additional outcomes also need careful definition and measurement and should be fully specified in the protocol, since extra resources often are needed to measure and evaluate them.

2.1.3. Other variables which are not study outcomes

Often, trials have other variables measured in the study not directly related to the study outcomes. Variables, such as age, gender, educational or socio-economic level, and nutritional status, may be used to evaluate potential effect modifiers or confounders to the study outcomes. These variables also need to be defined and considered at the beginning of the study, so they may be included in any pilot investigations.

2.2. Clinical case definitions

2.2.1. physician-based case definitions.

In some trials, outcomes are based on a clinical diagnosis by a physician, without any type of laboratory confirmation. For example, pneumonia may be diagnosed by auscultation in a trial evaluating the impact of an intervention designed to reduce indoor air pollution. This type of outcome is subjective, and interpretation may vary among doctors, and even among experienced specialists. Nevertheless, in many clinical trials, physician-based clinical diagnosis determines the main outcome of the study, since no alternatives exist. For many diseases, standardized criteria for defining a ‘case’ have been established by experts. The International Classification of Diseases ( World Health Organization, 2010 ; see also < http://www.who.int/classifications/icd/en >), which is revised about every 10 years, provides a basis for coding all diseases in a systematic way and is widely used for clinical and epidemiological research.

If standardized criteria for a ‘case definition’ have not been developed for the disease under study, a suitable definition should be established before the trial starts. For infectious diseases, there is often the need to distinguish between infection and disease, since clinical manifestations of infections may vary widely, from subclinical to overwhelming disease. For many trials, the main outcome of public health interest may be those infections that are severe or fatal. Careful definitions of these types of clinical categories are important, and, if available, the criteria used in other studies should be used to facilitate comparability across studies. The physicians charged with making diagnoses in the trial should discuss and agree the criteria they will be using to make a diagnosis and should compare their diagnoses on a range of patients prior to the start of the trial and at periodic intervals throughout the trial (see Section 2.2.5 ). Cases may also be classified as suspected, probable, or definite, using clinical and/or laboratory criteria.

In some populations, the conduct of a clinical examination may be problematic. Physical examinations are virtually always highly personal and may raise sensitive issues concerning individual dignity. In those populations when privacy is required, a third person in the examination room is often important, both to reassure the patient and to provide protection against possible charges of misconduct. In the case of children, the mother’s presence should normally be requested; for the examination of women, a nurse and an appropriate family member may be needed, even when the examiner is a woman. If there are local codes of behaviour that cover such circumstances, these must be adhered to.

2.2.2. Laboratory-based case definitions, including any diagnostic procedure

Commonly, a clinically defined study outcome involves the combination of a clinical assessment with the support of a confirmatory laboratory, or other diagnostic, procedure. For example, the clinical diagnosis of malaria may be supported by a positive identification of the parasite in the blood, or the diagnosis of dengue fever in a subject with 48 hours of elevated temperature with a positive immunoglobulin M or viral antigen present in the blood, as detected by polymerase chain reaction, or the clinical diagnosis of pneumonia with a confirmatory chest X-ray. All these diagnostic procedures need careful definition, including the technique, machine, or equipment to be used, reference values considered normal for the study population, and the level at which they will be considered abnormal. It is important to describe, in the protocol, how the test or procedure will be conducted and whether a reference laboratory will be used to validate the site laboratory or procedure—also, how procedures used by laboratory personnel to interpret results will be standardized and how monitoring for QC will be done. Some diagnostic results are also affected by subjectivity such as reading the results of a chest X-ray. In such cases, protocols have been developed to try to standardize the diagnosis, such as establishing defined criteria for each type of pathology in advance, having two independent, blinded radiologists read all X-ray films, with a third radiologist reading all films where there were disagreements, with their result used as the tiebreaker. Similar procedures have been developed to read blood smears for malaria. All these options have important consequences on the trial logistics and cost, so careful consideration needs to be given to them when designing the trial and selecting its study outcomes. Issues concerning laboratory tests of relevance to diagnosis in field trials are outlined in Chapter 17 .

2.2.3. Lay worker-based case definitions

Some trials use lay workers (fieldworkers) to measure a study outcome. Examples of such trials are diarrhoeal diseases where prevalent diarrhoea might be defined as three or more liquid or semi-liquid stools passed in a 24-hour period, as reported by the mother or the child’s caretaker to a fieldworker, or hygiene behaviours observed by fieldworkers in spot household checks during a hand-washing intervention trial. These types of outcomes are usually captured in questionnaires or study forms. Interviewing techniques and questionnaire design are discussed in Chapter 14 .

Fieldworkers may also measure a clinical indicator such as the body temperature or respiratory rate. Because of the high cost of using physicians, in many trials, lay workers or paramedical workers are trained to assess clinical signs and symptoms. When using lay workers or professional fieldworkers, such as nutritionists, auxiliary nurses, or nurse technicians, it is essential to train them and standardize the methods they use, in order to assure uniform implementation of these procedures in the field throughout the study, with good supervision and QC procedures.

2.2.4. Case definitions using secondary data sources

In some trials, such as in phase IV trials, existing surveillance systems may be used to define a study outcome. These secondary data sources, in which trial outcomes are not measured directly by study staff, will have the limitations intrinsic to the quality of the existing surveillance system. Examples of such study outcomes are post-marketing passive surveillance of vaccine or drug-related SAEs, such as hospitalizations of any type, after the introduction of the intervention into general use. They could also be used to evaluate the efficacy of a new vaccine or intervention on an important outcome which, for reasons of cost or ethics, could not be measured in a phase III trial such as the impact of a new vaccine on mortality.

2.2.5. Standardization

All study outcomes to be used in a clinical trial need to be properly standardized. When an outcome requires physicians, other professionals, or lay workers to measure it, standardization usually requires predefined exercises, with the use of an expert to act as the ‘standard’ against which the group is compared, defining differences which will be considered acceptable as part of the precision of the study. These standardization exercises could be done with real patients or mock subjects who may be trained actors. The use of videos showing different types of patients, which all participants evaluate independently, is a very useful exercise to help standardize them against the ‘standard’ observer. Standardization of this sort is not easy; it requires resources, time, and, in many cases, patients or volunteers willing to be examined by multiple persons. Ideally, the same set of samples, films, blood smears, subjects, or videos would be evaluated again by the same individual in a random order, under code, to allow the calculation of intra-observer reproducibility. All these procedures need to be carefully described in operating manuals and recorded, so they can be reviewed by investigators, collaborators, or regulatory agencies. In studies that last for several years, it is important to re-standardize observers every 6 to 12 months or if any observer needs to be replaced, to assure that the quality of the study is maintained.

2.2.6. Inclusion and exclusion criteria

An important component of an outcome definition is the description of the inclusion and exclusion criteria for the subjects to be evaluated in the trial. Ideally, the trial results should be able to be generalized to the whole population in which the intervention will be used. Under ideal circumstances, nobody should be excluded from the trial. However, for ethical, logistic, or analytical reasons, most trials establish stringent inclusion and exclusion criteria to exclude certain persons from participation. These criteria could be established on the basis of factors such as age, gender, literacy, being healthy or not, not affected by chronic diseases different from the study outcome, or not affected by other conditions such as abnormal baseline laboratory results. All these criteria need careful evaluation and discussion not only within the research team and the sponsor of the trial, but also with the ethics committees, the regulatory agencies overseeing the trial, and the communities in which the trial will take place, to assure that the trial results can be generalized to the intended population. It is common practice to exclude persons who are very sick from a trial (unless, of course, the trial intervention is directed at such persons). This is done because early deaths, or other SAEs in such persons, may occur independently of the trial intervention but may complicate interpretation of the effects of the intervention.

Signing a written informed consent form is now a standard inclusion criterion in most clinical trials (see Chapter 6 ). However, such a requirement will select a subgroup of the population who accept to sign such a form and participate in the study, generating a potential selection bias. To measure how strong that bias may be, it is important to register all eligible subjects who were considered as potential participants in the trial, indicating the reasons for refusal for those who did not enter into the trial.

2.3. Death and verbal autopsies

Preventing deaths (or severe disabilities) is one of the most important public health outcomes of any type of treatment or preventive intervention. It is the most important outcome in driving disease control policies and the introduction of new interventions or treatments into the population, once they have been found to be safe and effective. These types of outcomes have the heaviest weight in terms of disability-adjusted life-years (DALYs), when undertaking cost-effectiveness analyses of new drugs or interventions (see Chapter 19 ). Therefore, trials designed to evaluate these outcomes are very important. But, for many reasons, they may be difficult and costly to conduct, and, in many cases, they may not be feasible or ethical to do. Counting deaths in the conduct of a trial is a very sensitive issue, particularly in developing countries with poor health systems. It may create moral issues or generate political tension that may stop the trial. Therefore, few trials are done with these important outcomes, despite their major importance. However, those trials that are done with this endpoint and which demonstrate that an intervention significantly reduces mortality are most likely to influence a policy decision on a more widespread introduction of the intervention.

When deaths or severe disability are chosen as study outcomes, several problems emerge, depending on the setting where the study is conducted. In many LMICs, the quality of vital registration systems is poor or they are non-existent, precluding their use. Therefore, methods are needed to identify deaths, as well as to establish causes of death. In LMICs, the most commonly used method to ascertain causes of death are ‘verbal autopsies’. A verbal autopsy is a structured interview, conducted with the relatives of the deceased person, with the intention to reconstruct the series of events that led to the death (or severe complication or disability). Standard verbal autopsy questionnaires have been developed ( World Health Organization, 2012 ). Such ‘autopsies’ should be conducted neither too soon after the death (to avoid asking questions when relatives are still very upset by the death) nor too long after the death (to avoid recall bias). This interview is then analysed in a standardized way, either by physicians or using a computer algorithm, to classify the likely cause of the death, following a predefined set of criteria ( Lopez et al., 2011 ).

The reliability of verbal autopsy methods varies according to the cause of death, as some causes of death may be confused because signs and symptoms in the illness leading up to death may be similar. The usefulness of verbal autopsies is also dependent on the culture of the population under surveillance. It is essential to pilot-test the (translated) questionnaire to assure that appropriate local words are used to ascertain signs or symptoms of the causes of death.

In many populations, there could be a wide range of reasons why deaths may not be reported, and therefore special care should be taken to ensure that ascertainment is as complete as possible. This becomes crucial when the study outcome is death in the perinatal period, since an important proportion of live births that die in the minutes or hours after birth could be either missed or wrongly reported as stillbirths. In some trials, members of the study community may be hired as local informants to report any deaths. Other techniques include enumerating all members in a community and checking for the absence of any of them in frequently conducted cross-sectional surveys. Special attention should be paid to households for which all members are absent during one of these follow-up surveys, because the death of an adult may lead to dissolution of a household or migration of household members. Enquiries should be made with neighbours in such circumstances. Training and standardization of interviewers are essential. The frequency of surveillance will be a critical decision in designing trials with mortality outcomes, since a long recall period (such as 1 year) may miss deaths, particularly of children or infants; but each additional surveillance round will be expensive.

2.4. Non-clinical case definitions

Non-clinical case definitions can also be used in trials such as quality of life in trials of the use of chemotherapy for advanced cancer, antibiotic use in children in settings where they are available without prescription, satisfaction of users of a health service, and economic outcomes (costs) which are discussed in Chapter 19 . They also may include outcomes that come directly from patients about how they feel or function in relation to a health condition and its therapy (so called patient-reported outcomes ), without interpretation by health care professionals or anyone else. For these case definitions, instruments that have been developed previously or that are created especially for the trial need to be validated, in order to have valid and comparable results.

2.5. Proxy measurements as study outcomes

Some trials may select outcome measures that are associated with the outcome of interest such as reported risky sexual behaviour, which are either easier to measure, cheaper, or more socially acceptable. Those outcomes are called ‘proxy’ measurements of the outcome of interest. Such measures, however, may be subject to invalidity and bias (for example, misreporting, differential degrees of desirability bias between trial arms).

2.5.1. Behavioural changes

A behaviour thought to be critical to reduce the disease of interest might be selected as a study outcome. For example, in a study to investigate the effectiveness of a health education campaign to promote the use of latrines, where the ultimate objective was to reduce diarrhoeal disease, the frequency of use of latrines might be measured. Sometimes, health-related behaviours may be measured by direct observation.

Changes in knowledge or attitudes are sometimes an important initial step before a behaviour is changed, which, once changed, should reduce the risk of the disease of interest. Knowledge or attitudes can be assessed with reasonable reliability, using questionnaires or other interview methods, but observational studies may be required to determine if behavioural changes have actually occurred. For example, in a study to investigate the effectiveness of a health education campaign to promote the use of latrines, it may be relatively straightforward to assess, after the campaign, whether individuals have a better knowledge of why using latrines is desirable, but observational studies, before and after the campaign, may be necessary to ascertain whether or not the frequency of use of latrines had actually changed, let alone whether behavioural change led to a reduction in the incidence of diarrhoea. Similar issues arise with respect to the evaluation of a hand-washing intervention campaign. Further studies may then be needed to determine whether the changed behaviour has led to a reduction of diarrhoeal diseases.

Some trials have the incidence of a self-reported behaviour as one of their outcomes. For example, in evaluating the effectiveness of sexual behaviour change interventions, it is not possible to observe sexual behaviours directly, so self-reported behaviours are frequently recorded. But such measures are very open to desirability bias where the respondent reports the behaviour that they think the investigator would judge to be the desirable one. Furthermore, the desirability bias may be differential between the trial arms. For example, if the intervention group has been encouraged to reduce their number of sexual partners and always use a condom, while the control group has not, the intervention group may be more likely to over-report these ‘desired’ behaviours at follow-up. Self-reported behaviours, though sometimes the only practical outcome for a trial, are potentially misleading and should be avoided, at least as the primary outcome measure in a trial, if at all possible.

2.5.2. Transmission reduction

The purpose of interventions, based on vector control or environmental alteration, may be to reduce or interrupt transmission of the infectious agent of interest. Generally, the first priority is to determine whether the intervention has accomplished the immediate changes intended. For example, in trials in which insecticides are applied to reduce vector populations in order to reduce the transmission of some infectious agent, the first step would be to determine the impact of the intervention on the vector population. If the vector population is little affected, it may be reasonable to conclude that any impact on human disease is unlikely. However, if there is a reduction in vector population, it may be erroneous to conclude that the human disease load will also fall. A further study to determine the impact on disease may be required. Similarly, if interventions are being evaluated that may reduce indoor air pollution as a measure against respiratory disease, it may be best to focus initial studies on the assessment of changes in pollution levels, before assessing the impact on respiratory diseases. Usually, it will be more efficient to carry out trials to monitor the impact on disease only after there is evidence of an effect on the vector or on the agent against which the intervention is directed.

incidence of infection or disease

prevalence of infection or disease

severity of disease

intensity of infection (for example, for helminths)

intensity of infective agent in the vector.

Any changes to these different outcomes will happen at different intervals after the intervention is in place, and may require studies over time to measure the overall study impact. For instance, in an onchocerciasis control programme, the first evidence that an intensive larviciding of Simulium damnosum (black fly) breeding sites is having an effect may be a dramatic drop in fly-biting rates in the intervention area. Over the next several years, there may be a steady fall in the intensity of microfilarial infections among those living in the endemic area, but only after some years might it be possible to detect evidence of a fall in the prevalence of infection, and later still an impact on blindness rates which is the major adverse health consequence of onchocercal infection.

2.6. Adverse events

An important outcome of all trials is to assess the safety of the intervention under evaluation (for example, of a new drug or vaccine). Adverse events (AEs) are defined as any untoward clinical or laboratorial medical occurrence in a patient or clinical investigation subject, related or not to the use of an intervention in a trial. Serious AEs (SAEs) are defined as any events that are life-threatening or result in death. They include patient hospitalization or prolongation of existing hospitalization, events that result in persistent or significant debilitation or incapacity, and congenital anomalies and birth defects. All SAEs should be reported immediately to the sponsor (or the DSMB on behalf of the sponsor), followed by detailed written reports (see Chapter 7 ). Usually, two types of study outcomes are defined: (1) the active, prospective evaluation of a set of predefined potential AEs known or suspected to be associated with the type of drug, vaccine, or product under evaluation, and (2) recording all clinical or laboratory abnormalities, expected or not, that occur in study subjects during a specified time period or throughout the conduct of the trial, by active or passive surveillance, which may reveal an adverse consequence previously not known to occur with the drug, vaccine, or product under evaluation. For both types of safety outcomes, criteria must be developed to assess the severity, as well as the incidence of AEs associated with the drug, vaccine, or product under evaluation. Severity can be measured by the magnitude of a laboratory or clinical test abnormality, or by the subjective perception on how much the AE altered the function or quality of life of the individual. For instance, a reaction at the site of injection of a vaccine could be graded as mild if only a colour change is noted with mild pain, without induration and without any restriction on the arm or leg movement; moderate if, in addition to colour change of the skin, induration is noted and there is some restriction of movement; and severe if the subject cries out or winces if the area is touched and the arm or leg cannot be moved without pain. In many studies, a diary card may be provided to the study subject or, in case of children, to the mother or caretaker to record these reactions during a 7- or 14-day period after the administration of a vaccine or during the drug therapy. To aid measuring an injection site reaction, a ruler may be provided to the subject. And to standardize the measurement of temperature, a digital thermometer may be provided as well. Study subjects or children’s mothers or caretakers need to be appropriately trained in using these study cards and instruments. In addition to its severity, these reactions are usually classified as unrelated, unlikely to be related, or possibly related to the intervention under evaluation. The criteria used for this classification may include proximity of the event to the administration of the intervention (for instance, a rash developing within 20 minutes of an injection would most likely be classified as possibly related), the unusualness of the clinical event (a disease which normally occurs in that age group or a complication expected to happen in the disease under study), or even the subjective interpretation of the investigator. Whatever criteria are used should be stated. The incidences of AEs, graded by the severity and likelihood of being related to the interventional product, are later compared between the study group exposed to the intervention and the control group (using placebo or an active comparator) to assess statistically if AEs of different kinds were or were not associated with the drug, vaccine, or product.

All safety measurements need careful definition in the study protocol, study forms to record them, using standardized measurements and codes to register them, and active monitoring of their occurrence. Most trials require those AEs that are considered serious to be individually reported to the sponsor and to an ethics review board, to the regulatory agency overseeing the trial, and to an independent DSMB for their careful evaluation during the conduct of the trial, to allow the possibility for the trial to be stopped or modified before its completion if it is suspected that SAEs are associated with the drug, vaccine, or product under investigation.

3. Factors influencing choice of outcome measures

The choice of the outcome measures in a specific trial largely depends on the purpose of the trial and how relevant, feasible, and acceptable the measures will be in a particular study population. Furthermore, the choice may be constrained by economic, logistic, or ethical considerations.

3.1. Relevance

Interventions are generally designed to reduce disease and/or to promote health. The outcome measures chosen should reflect these objectives as fully as possible, but, when intermediate variables are used, rather than those of main interest, care must be taken to choose variables of direct relevance to the main outcome. This is not always straightforward. For example, it may be decided to assess the impact of a vaccine by measuring the proportion of individuals who develop antibodies to the vaccine. This may be reasonable if it is known that there is a high correlation between the development of antibodies and protection from clinical disease. For many diseases, however, this relationship has not been established, and it would not be warranted to base conclusions regarding protection against disease simply on antibody determinations.

A health education intervention may be designed to change behaviour to reduce disease risk, but, as discussed in Section 2.5.1 , asking individuals if they have changed their behaviour may give a measure of impact that correlates poorly with true changes in the risk of disease. Are individuals responding truthfully? Are they doing what they say they do? Even if behaviour changes, is this associated with a lowering in the incidence of disease?

The outcome variable measured should be as close as possible to the outcome of main interest. While this may seem an obvious suggestion, it may have major impact on the design of a study. For example, if the prevention of death is of prime interest, then, whenever possible, this should be made the endpoint of the trial. To do so might require an increase in the size of the trial from hundreds to thousands, or even tens of thousands, of individuals. Such a large trial might be difficult to find funding for, and there may never be an adequate test of whether the intermediate variables measured are acceptable surrogates for effects on mortality.

3.2. Feasibility

To be successful, a trial must be designed to have achievable objectives. A trial which has mortality as the endpoint, but which is too large to be successfully completed, may be of less value than a well-designed smaller trial aimed at assessing the impact on some intermediate endpoint such as severe disease. There must often be a compromise between relevance and feasibility. It is pointless to set unachievable goals, even if they look attractive in the objectives section of a proposal. Also, it may be of little value to measure the effect of an intervention on an outcome measure which is only distantly related to the measure of prime interest. The outcome measures selected will be much influenced by the resources available for the trial, the availability of skilled personnel, and the necessary laboratory support to diagnose cases of disease. In many large trials, every individual in the study population may have to be screened for disease or infection in a relatively short time. With such time constraints, some individuals may be misdiagnosed. The consequences of reductions in diagnostic sensitivity and specificity are discussed in Section 4.2 .

3.3. Acceptability

The acceptability of the measurement of an outcome variable to the study population is critical to the successful conduct of a trial. For example, the recording of birthweights may not be possible in a population that allows only close relatives to have access to a mother for a few days or weeks after the child’s birth. Taking venous blood samples or repeated blood samples is unpopular in many societies. If the method for measuring the outcome involves pain or inconvenience to the participants, it may be necessary to modify or abandon it. An outcome, of which the assessment involves a long interview with participants at a time when they would otherwise be planting crops or taking care of their household chores, may be unacceptable; it may either have to be abbreviated or carried out at a more convenient time.

3.4. Opportunity for add-on studies

Some trials offer the opportunity to measure outcomes that are not directly related to the objectives of the original study itself. These opportunities can be exploited by researchers to answer questions with minimal additional funding. For example, a diarrhoeal surveillance study might be carried out within a clinical trial in which a cohort of healthy children is being followed over time. However, it is very important that the add-on study does not interfere with the original study outcome measure. Such additions should be considered at the beginning of the study and should have a separate study protocol. It is also important to inform sponsors, participants, and all stakeholders of the original trial of the coexistence of the proposed add-on study. Such investigations will usually require separate ethical approval and informed consent.

4. Variability and quality control of outcome measures

4.1. reproducibility.

The extent to which different observers will make the same diagnoses or assessments on a participant and to which observers are consistent in their classifications between participants may have an important influence on the results of a trial. Clearly, it is desirable to choose outcome measures for which there is substantial reproducibility and agreement among observers, with respect to the classification of participants in the trial.

For objective outcome measures, variations between observers, or by the same observer at different times, may be small and unlikely to influence the results of a study. For outcome measures requiring some degree of subjective assessment, however, such variations may be substantial. The likely degree of such variations will influence the choice of outcome measures, as it will be preferable to select those measures that have the smallest inter- and intra-observer variations, yet still give valid measures of the impact of the intervention.

Variation among observers is often much greater than expected, for example, in the reading of a chest X-ray to assess whether there is evidence of pneumonia. If a study involves several observers, pilot studies should be conducted, in order to measure the extent of the variation and then to seek to standardize the assessment methods to minimize the variation. With suitable training, it is usually possible to reduce the variation between observers substantially.

For some outcomes, independent assessment by two observers should be routine, with a third being called in to resolve disagreements. It may be costly to screen the whole trial population in this way, but a common approach is to have all suspected cases of the disease of interest examined by a second observer, mixed in with a sample of those not thought to have the disease. Sometimes, it is possible to have the observer examine the same individual twice, but these examinations may not be independent, unless the survey is large and the observer does not remember the result of the first assessment.

It is important to make every effort to reduce variability to the maximum extent possible. Having done so, however, it is also critical to know the extent of the remaining ‘irreducible’ variability for purposes of analysis. The purpose of trials is usually to demonstrate the effect of an intervention or to compare differences between interventions. Knowledge of the inherent variability in diagnostic procedures is essential for this demonstration, and the best way of assessing this is through replicate measures. It is especially important to take account of between-observer differences when communities are the units of randomization in a field trial. Differences between observers may produce biases if different observers are used in different communities. In such situations, it is better to organize the fieldwork so that the workload within each community is split among different observers and differences between the observers are not confounded with the effect of the intervention.

4.2. Sensitivity and specificity

The choice of an appropriate definition of a ‘case’ in a field trial will be influenced by the sensitivity and specificity associated with the diagnostic criteria. Sensitivity is defined as the proportion of true cases that are classified as cases in the study. Specificity is the proportion of non-cases that are classified as non-cases in the study. A low sensitivity is associated with a reduction in the measured incidence of the disease. This decreases the likelihood of observing a significant difference between two groups in a trial of a given size. In statistical terms, it reduces the power of the study (see Chapter 5 , Section 2.2 ). If the incidence of the disease in both the intervention group and the comparison group will be affected proportionately in the same way, as is often the case, it does not bias the estimate of the relative disease incidence in the two groups, though the absolute magnitude of the difference will be less than the true difference. Thus, in the context of a vaccine trial, because protective efficacy is assessed, in terms of relative differences in incidence between groups, the estimate of protective efficacy will not be biased, but the confidence limits on the estimate will be wider than they would be using a more sensitive case definition. In theory, the reduction in power associated with low sensitivity can be compensated for by increasing the trial size.

In general, a low specificity of diagnosis is a more serious problem than a low sensitivity in intervention trials. A low specificity results in the disease incidence rates being estimated to be higher than they really are, as some participants without the disease under study are classified incorrectly as cases. Generally, the levels of inflation in the rates will be similar, in absolute terms, in the intervention and comparison groups, and thus the ratio of the measured rates in the two groups will be less than the true ratio, though the difference in the rates should be unbiased. Thus, in vaccine trials, for example, the vaccine efficacy estimate will be biased towards zero, though the absolute difference in the rates between the intervention and control groups will not be biased (unless there is also poor sensitivity). Increasing the trial size will not compensate for the bias in the estimate of vaccine efficacy.

In algebraic terms, suppose the true disease rates are r 1 and r 2 in the two groups under study, the true relative rate R is r 1 − r 2 , and the true difference in disease rates D is r 1 / r 2 . If sensitivity is less than 100% (but specificity is 100%), and only a proportion k of all cases are correctly diagnosed, the measured disease rates in the two groups will be k r 1 and k r 2 ; the measured relative rate will be k r 1 / ( k r 2 ) = R ; and the measured difference in disease rates will be k r 1 − k r 2 = k ( r 1 − r 2 ) = k D (which will be less than D ). If specificity is less than 100% (but sensitivity is 100%), and the rate of false diagnoses is s , the measured rates in the two groups will be ( r 1 + s ) and ( r 2 + s ) ; the measured relative rate will be ( r 1 + s ) / ( r 2 + s ) (which will be less than R ); and the measured difference in disease rates will be ( r 1 + s ) − ( r 2 + s ) = D .

To measure the sensitivity and specificity of the diagnostic procedures used in a trial, it is necessary to have a ‘gold standard’ for diagnosis (i.e. it is necessary to have a diagnostic procedure that determines who really is a case and who is not). Sometimes, this is not possible, and, even if definitive diagnostic procedures exist, it may be necessary to use imperfect procedures in a field trial for reasons of cost or logistics. In this situation, if an assessment is made of sensitivity and specificity, it is possible to evaluate the consequences for the results of a field trial, and possible even to correct for biases in efficacy estimates due to the use of a non-specific diagnostic test. Unfortunately, in many situations, there is no ‘gold standard’, and so the sensitivity and specificity of the diagnostic methods used remain uncertain. For example, there is no universally agreed definition of a case of clinical malaria. Most would agree that the presence of parasites in the blood is necessary (unless a potential case has taken treatment before presenting to the study clinic), and many would agree that the presence of fever associated with parasitaemia increases the likelihood of the disease being clinical malaria, but it is also possible that the fever is due to other causes, rather than the parasitaemia being the cause of the fever.

The bias induced by a low specificity of diagnosis is most severe for diseases that have a low incidence. A good example of this is provided by leprosy, which is both difficult to diagnose (in the early stages) and also of low incidence. Consider a vaccine trial in which the true disease incidence in the unvaccinated group is ten per thousand over the period of the trial, and the true efficacy of a new vaccine against leprosy is 50%, i.e. the true disease incidence in the vaccinated is five per thousand over the period of the trial. If the sensitivity of the diagnostic test used for cases is 90%, but the specificity is 100%, the observed disease incidences would be 10 × 0.9 = 9.0 and 5 × 0.9 = 4.5 per thousand, respectively. Thus, the estimate of vaccine efficacy is correct (50%). The power of the study is reduced, however. To achieve the power that would be associated with a ‘perfect’ test, the trial size would have to be increased by about 11%.

On the other hand, if the specificity of the diagnostic test is as high as 99% and the sensitivity is 100%, the observed disease incidences would be ten true cases + (990 × 0.01 = 9.9) false cases = 19.9 per thousand in the unvaccinated group, and five true cases + (995 × 0.01 = 9.95) = 14.95 per thousand in the vaccinated group. Thus, even with a test with 99% specificity, the estimate of vaccine efficacy is reduced from the true value of 50% to 25%. If the specificity of the test were 90%, the expected estimate of vaccine efficacy would be only 4%.

In vaccine trials, the sensitivity and specificity of the diagnostic test are of consequence in different ways at different times in the trial. When individuals are screened for entry to the trial, it is important that the test used should be highly sensitive, even if it is not very specific, as substantial bias may be introduced if undiagnosed ‘cases’ are included in the trial and included in the vaccinated or unvaccinated groups. If the vaccine has no effect on the progression of their disease and they are detected as cases later in the trial, a false low estimate of efficacy will result. Thus, individuals whose diagnosis is ‘doubtful’ at entry to the trial should be excluded from the trial. Conversely, once individuals have been screened for entry into the trial and they are being followed for the development of disease, a highly specific test is required to avoid the bias illustrated in the preceding paragraph.

In situations where there may be no clear-cut definitions of a case (for example, early leprosy or childhood TB), studies of intra- and inter-observer variation may be undertaken, using various definitions of the disease. The definition that shows the least disagreement between observers and gives maximum consistency within each observer may be the appropriate one to use in a trial, but the investigator should be aware of the potential for bias if the specificity of the diagnostic procedure is less than 100%.

The most powerful way to minimize bias in the assessment of the impact of an intervention is through the conduct of a double-blind randomized trial. If these two aspects are built into a trial, an effect of an intervention is not likely to be observed if there is no true effect. However, as pointed out in Section 4.2 , if the specificity of the diagnosis for the outcome of interest is poor, the estimate of the efficacy of an intervention, measured in relative terms, may be biased towards zero, even in a properly randomized double-blind investigation.

It is highly desirable that the person making diagnoses in a trial is ignorant of which intervention the suspected cases have received. If the diagnosis is based on laboratory tests or X-ray examinations, blindness should be easy to preserve. In some circumstances, it may be possible to determine from the results of a laboratory test which intervention an individual has received, as the test may be measuring some intermediate effect between the intervention and the outcome of prime interest (for example, an antibody response to a vaccine). In such cases, those making diagnoses in the field should not be given access to the laboratory results. For example, in placebo-controlled studies of praziquantel against schistosomiasis in communities where the infection is common, those who had received the active drug would be easily detected by a rapid reduction in egg counts in stool or urine samples following treatment. If the outcome of main interest is morbidity from the disease, then the egg count information should be kept from those making the assessment of morbidity. It would generally be inappropriate to use measures of antibody level to make diagnoses of disease following vaccination, if the vaccination itself induced antibodies indistinguishable from those being measured. Similarly, tuberculin testing should not be part of diagnostic procedures for TB in studies of the efficacy of BCG vaccination, as the vaccine alters the response to the test.

If the diagnosis of disease is based on a clinical examination, it may be necessary to take special precautions to preserve blindness. An example is given in Chapter 11 , Section 4 , with respect to a BCG trial against leprosy, in which all participants had the upper arm area, where BCG or placebo was injected, covered during the clinical examination, since BCG leads to a permanent scar. Even if the participants know which intervention they had, it is important to try to keep this knowledge from the person making any diagnoses. Thus, participants might be instructed not to discuss the intervention with the examiner, and the examiner would be similarly restricted. Such a procedure is obviously not fail-safe, but great efforts should be made to preserve blindness, if at all possible, especially if the diagnosis is made on subjective criteria.

If randomization in a trial is by community, rather than by individuals, it may be especially difficult to keep examiners ignorant of the intervention an individual received. Sometimes, ways can be found of doing this, for example, by conducting surveys for disease by bringing all participants to a clinic outside the trial communities. If communities are randomized to receive an improved water supply or not, one outcome measure of interest might be the incidence of scabies infection. It may be difficult to avoid the possibility of the diagnoses of scabies being influenced by the observer’s knowledge of whether or not the participant was in a village with an improved water supply. In such a case, it may be best to seek other measures of impact, based upon objective criteria or laboratory measures, or to take photographs of the relevant body parts and have these assessed objectively and ‘blind’ to intervention group.

4.4. The Hawthorne effect

Trials that require active home visits by study personnel during the surveillance period to evaluate the effect of an intervention may be affected by an indirect effect of the home visits on the study objective, even when not intended. The presence of a study member in a subject’s home may have a positive effect on the health status of the subject, since it may, for example, stimulate better health behaviour of the subject or improve hygiene practices in the house or better health care utilization. In studies with such effects, rates of illnesses or of severe illness may be reduced in both study arms—an indirect effect known as the ‘Hawthorne effect’ (named after a study in the 1930s in the USA at the Hawthorne Works, in which it was documented that worker behaviour changed as a consequence of them being observed). This effect reduces the power of the study and may make it inconclusive. There is no easy way to control for it, so, if such a Hawthorne effect is expected in a field trial, the sample size may need to be increased to maintain statistical power.

4.5. Quality control issues

The sensitivity and specificity of the diagnostic procedures employed in a trial should be monitored for the duration of the trial, as they may change as the study progresses. Such changes may be for the worse or for the better. With experience, diagnostic skills may improve, but also, as time passes, the staff may become bored and take less care. It is important that the field staff are aware that their performance is being continuously monitored. If this is done, then anyone who goes ‘off the rails’ can be steered back or removed from the study, before much harm is done. Such monitoring is important for both field and laboratory staff.

The methods used to monitor the quality of diagnostic procedures may include the re-examination of a sample of cases by a supervisor or a more highly trained investigator and, for the laboratory, may be done by sending a sample of specimens to a reference laboratory and by passing some specimens through the laboratory in duplicate, in a blinded fashion, to determine if the differences between results on the same specimen are within acceptable limits (see Chapter 17 , Section 5 ).

If the disease under study is relatively rare, it may be difficult to measure sensitivity based on small numbers of individuals being examined twice. While it will be possible to check if specificity is poor (a high proportion of those classified as cases are wrongly diagnosed), checks on sensitivity may involve the examination of thousands of individuals twice to determine if cases are being missed. Fortunately, in most trials, specificity is of more critical importance than sensitivity, although the relative importance can change as the survey goes on, as discussed in Section 4.2 .

• Cutts, F. T., Zaman, S. M., Enwere, G., et al. 2005. Efficacy of nine-valent pneumococcal conjugate vaccine against pneumonia and invasive pneumococcal disease in The Gambia: randomised, double-blind, placebo-controlled trial. Lancet , 365 , 1139–46.10.1016/S0140-6736(05)71876-6 [ PubMed : 15794968 ] [ CrossRef ]
• Lopez, A. D., Lozano, R., Murray, C. J. L., Shibuya, K. 2011. Verbal autopsy: innovations, applications, opportunities improving cause of death measurement. Population Health Metrics , 9 , 128–254.
• Ross, D. A., Changalucha, J., Obasi, A. I., et al. 2007. Biological and behavioural impact of an adolescent sexual health intervention in Tanzania: a community-randomized trial. AIDS , 21 , 1943–55.10.1097/QAD.0b013e3282ed3cf5 [ PubMed : 17721102 ] [ CrossRef ]
• Schellenberg, D., Menendez, C., Kahigwa, E., et al. 2001. Intermittent treatment for malaria and anaemia control at time of routine vaccinations in Tanzanian infants: a randomised, placebo-controlled trial. Lancet , 357 , 1471–7.10.1016/S0140-6736(00)04643-2 [ PubMed : 11377597 ] [ CrossRef ]
• World Health Organization. 2010. International statistical classification of diseases and related health problems , 10th revision, Volume 2, instruction manual [Online]. Geneva: World Health Organization. Available at: < http://www ​.who.int/classifications ​/icd/ICD10Volume2_en_2010 ​.pdf >.
• World Health Organization. 2012. Verbal autopsy standards: the 2012 WHO verbal autopsy instrument [Online]. Geneva: World Health Organization. Available at: < http://www ​.who.int/healthinfo ​/statistics ​/WHO_VA_2012_RC1_Instrument.pdf >.

This is an open access publication. Except where otherwise noted, this work is distributed under the terms of the Creative Commons Attribution NonCommercial 4.0 International licence (CC BY-NC), a copy of which is available at http://creativecommons.org/licenses/by-nc/4.0/ . Enquiries concerning use outside the scope of the licence terms should be sent to the Rights Department, Oxford University Press, at the address above.

Monographs, or book chapters, which are outputs of Wellcome Trust funding have been made freely available as part of the Wellcome Trust's open access policy

• Cite this Page Smith PG, Morrow RH, Ross DA, editors. Field Trials of Health Interventions: A Toolbox. 3rd edition. Oxford (UK): OUP Oxford; 2015 Jun 1. Chapter 12, Outcome measures and case definition.
• PDF version of this title (6.4M)

In this Page

• Introduction to outcome measures and case definition
• Types of outcome measures
• Factors influencing choice of outcome measures
• Variability and quality control of outcome measures

Other titles in this collection

• Wellcome Trust–Funded Monographs and Book Chapters

Related Items in Bookshelf

• All Methods Resources
• All Monographs
• All Textbooks

Related information

• PubMed Links to PubMed

Recent Activity

• Outcome measures and case definition - Field Trials of Health Interventions Outcome measures and case definition - Field Trials of Health Interventions

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 10 June 2024

A cross-sectional National Health and Nutrition Examination survey-based study of the association between systemic immune-inflammation index and blood urea nitrogen levels in United States adolescents

• Cheng Guo 1   na1 ,
• Qinhui Cai 2   na1 ,
• Yang Li 1 ,
• Feng Li 1 &
• Kai Liu 1  

Scientific Reports volume  14 , Article number:  13248 ( 2024 ) Cite this article

186 Accesses

Metrics details

• Endocrine system and metabolic diseases

Blood urea nitrogen (BUN) level is one of the commonly used indicators to assess renal function and systemic immune-inflammatory status. In the adolescent population, changes in BUN levels may be associated with a variety of factors, including physiologic dehydration, lifestyle influences such as nutritional intake, physical activity, and possible endocrine or metabolic disorders. In recent years, more and more studies have shown that BUN levels are not only a reflection of kidney function, but it may also be related to the inflammatory state of the body. The Systemic Immune Inflammatory Index (SII) is a comprehensive index that takes into account platelet counts, neutrophil and lymphocyte counts, and is thought to be effective in reflecting the body's immune status and inflammatory response. However, research on the relationship between the two, SII and BUN, remains understudied in the adolescent population. The purpose of this study was to examine the relationship between SII and BUN levels in a population of American adolescents and to further analyze the factors that influence it. We conducted a cross-sectional study using data from the National Health and Nutrition Examination Survey (NHANES) database. Using descriptive statistics, correlation analysis, and regression analysis, we explored the relationship between SII and BUN levels. We found a significant negative correlation between SII and BUN levels, with BUN levels decreasing when SII levels increased (BUN as the dependent variable and SII as the outcome variable). We performed a multiple regression analysis of this relationship, controlling for possible confounders such as gender, age, race, and BMI, and found that this negative correlation remained significant. Our findings reveal an important relationship between SII and BUN levels and provide new perspectives for understanding adolescent health.

Similar content being viewed by others

Age- and sex-specific reference intervals for blood urea nitrogen in Chinese general population

Association between systemic immunity-inflammation index and hypertension in US adults from NHANES 1999–2018

Elevated serum uric acid is associated with infertility in women living in America

Introduction.

In the field of medicine, an in-depth study of physiologic indicators in adolescents is key to understanding and promoting their healthy development. During adolescence, individuals experience rapid growth and changes in hormone levels in the body that have profound effects on the body's metabolism and immune system 1 , 2 , 3 , 4 , 5 . Of these metrics, Blood urea nitrogen (BUN) and Systemic Immune Inflammatory Index (SII) are particularly important. BUN is commonly used to assess renal health and metabolic status 6 , 7 , 8 , whereas SII combines neutrophil, lymphocyte, and platelet counts and is thought to provide a comprehensive picture of an individual's immune and inflammatory status 9 , 10 , 11 , 12 , 13 . The importance of BUN and SII is widely recognized in adult health research, but questions about these two metrics how they interact and influence each other in the adolescent population is relatively limited. The physiological characteristics of adolescence differ significantly from those of adulthood, which prevents the simple application of adult findings to adolescents 14 , 15 . For example, maturation of the urinary and immune systems during adolescence may affect BUN and SII levels and their correlations in different ways 16 , 17 , 18 , 19 . Furthermore, although several studies have examined the relationship between SII and BUN levels, the results of these studies have been inconsistent. Some studies have found a positive correlation between SII and BUN levels, whereas others have found a non-significant or negative correlation 20 , 21 , 22 , 23 , 24 . These contradictory results may be due to differences in research methods such as study design, sample selection, statistical methods, or different age, gender, ethnicity, lifestyle, and environmental factors that may have an impact on the relationship between SII and BUN levels.

Therefore, the aim of this study was to investigate the interconnection between two indicators, SII and BUN levels, in a population of American adolescents. Since adolescence is a critical period of growth and development, with both kidney function and the immune system in developmental flux, theoretically, as kidney function improves, BUN levels may decrease if all other conditions remain constant. However, this change interacts with changes in protein metabolism during adolescence. If protein breakdown is increased and urea production is accelerated, BUN levels may remain unchanged or elevated despite normal renal function. SII is a composite index that includes platelet count, neutrophil count, and lymphocyte count. Puberty is an important period of adaptation for the immune system, and immune cell response and regulation may be altered by hormonal changes. For example, fluctuations in sex hormones are thought to be associated with immune regulation and may affect immune cell ratios and activity, thereby affecting SII values. As immunity matures, changes in the proportion of lymphocytes in the overall leukocyte population, for example, may result in higher or lower SII. In summary, developmental changes in kidney function and the immune system during adolescence directly affect BUN and SII levels. These changes reflect the complexity and variability of physiologic maturation. Understanding the relationship between BUN and SII is of great value in revealing possible health problems during this period.

We conducted a cross-sectional study using data from the National Health and Nutrition Examination Survey (NHANES) database, a nationally representative health and nutrition survey conducted by the U.S. Centers for Disease Control and Prevention (CDC), which includes data from a variety of clinical examinations, laboratory tests, and questionnaires. Our research hypothesis was that there was a significant correlation between SII and BUN levels and that this relationship would remain after controlling for other variables.

The innovation and contribution of this study is that most of the existing studies have focused on the adult population and relatively few studies have been conducted on the specific age group of adolescents. Focusing on the adolescent population, this study explored the association between BUN and SII and further analyzed the influencing factors. The results of this study may be of potential value in the management of adolescent health and in the early identification and prevention of heart- or kidney-related chronic diseases during adolescence. We look forward to more studies in the future that will build on our study and further explore this area. We believe that this innovative finding has important implications for the study of adolescent health.

Study population and data sources

NHANES is a representative U.S. national population survey that uses complex, multistage, and probability sampling methods to provide a wealth of information about the nutrition and health of the general U.S. population 25 .

The Ethics Review Board of the National Center for Health Statistics (NCHS) approved the study protocol. Written informed consent was obtained from all survey participants or parents and/or guardians of individuals under 18 years of age. Visit https://www.cdc.gov/nchs/nhanes/index.htm for additional information.

Data from the NHANES 2009–2018 consecutive cycles (N = 49,693) were selected for this study to assess the relationship between BUN and SII among adolescents. First, adolescents and young adults were defined according to the World Health Organization's 26 , 27 , 28 , 29 , 30 , 31 . A total of 8646 eligible adolescents aged 10–19 years were included. We excluded participants with missing BUN values (N = 2997) and missing SII values (N = 384) from the eligible participants. The study ended up with 5,265 participants who met the inclusion criteria for subsequent analysis. A detailed flow chart of the participant nativity criteria is shown in Fig.  1 .

Variables in research

Venous blood was drawn from participants, and serum samples were processed, stored, and then transported to a collaborative laboratory service department for analysis. The BUN was quantitatively determined using the enzyme conductance method (DxC800 biochemistry analyzer). The complete blood count was measured by professionally trained hematological researchers using an automatic hematology analyzer (Coulter DxH 800 analyzer). Subsequently, we conducted an analysis using lymphocyte, platelet, and neutrophil counts (expressed in thousands of cells/μL). Finally, SII (1000 cells/ul) was calculated as platelet count × neutrophil count/lymphocyte count as an exposure variable 32 . The NHANES Laboratory/Medical Technician Procedure Manual (LPM) details the collection and processing of specimens in exhaustive detail.

Other covariates included sex (male or female), race (Mexican American, other Hispanic, non-Hispanic black, non-Hispanic white, or other race) and education level (less than high school, or high school), and annual household income status. Continuous covariates included BMI (body mass index), RBC (red blood cells), HGB (hemoglobin), AST (aspartate aminotransferase), ALT (alanine aminotransferase), TG (triglycerides), TC (total cholesterol), Scr (serum creatinine), GLU (serum glucose), HbA1c (glycated hemoglobin), and 25-OH-D3. Every measurement procedure is accessible to the public via the following link: https://wwwn.cdc.gov/nchs/nhanes/Default.aspx

Methods of statistics

Every analysis used EmpowerStats software (version 4.1, http://www.empowerstats.com ) and R (version 3.4.3, http://www.R-project.org ). The mean by which standard deviation (S.D.) was calculated for continuous variables, while percentages were used to represent categorical variables. Analyses were conducted on the baseline characteristics of categorical and continuous variables utilizing chi-square tests and linear regression models, respectively. We categorized SII into four groups. After adjusting for potential confounders, we developed multivariate logistic regression models to explore independent associations between BUN and SII. Subgroup analyses were also conducted by gender, age, race, education, and body mass index. Three models were developed: an unadjusted model, a partially adjusted model (adjusted for age, gender, and race), and a fully adjusted model (adjusted for age, gender, race, education level, annual household income, BMI, RBC, HGB, AST, ALT, TG, TC, Scr, GLU, HbA1c, and 25-OH-D3). Values of P < 0.05 were considered to be statistically significant.

Ethics statement

This study was reviewed and approved by the NCHS Ethics Review Board. The patients/participants provided their written informed consent to participate in The patients/participants provided their written informed consent to participate in this study.

Population-specific baseline data

Weighted baseline characteristics of the 5265 included participants (2188 males and 2170 females) based on SII quartiles are shown in Table 1 . The mean age of the included participants was 15.45 ± 2.26 years, and the mean value of BUN was 10.99 ± 3.43 mg/dL. The ranges of SII quartiles 1–4 were < 285.42, 285.42–402.16, 402.16–560.25, and ≥ 560.25. BUN levels were significantly higher in the highest quartile of SII index participants than in the lowest quartile of SII participants, and similarly, the number of people of different ages, genders, races, and educational levels was significantly higher in the highest quartile of SII index participants than in the lowest quartile of SII participants; Scr, HGB, ALT, AST, Scr, and HbA1c levels were all significantly lower than the lowest quartile of SII participants, but TG levels were significantly higher than the lowest quartile of SII participants (all P < 0.05).

Table 2 shows the relationship between BUN and SII. We found that the lower the BUN level, the higher the SII value in both the original and adjusted model 1. In the fully adjusted model it was found that for every unit decrease in BUN levels, SII increased by 2.29 units (β = − 2.29, 95% CI − 4.41, − 0.16). In addition, we further characterized an inverse relationship between BUN and and SII (P for trend = 0.1710).

Table 3 shows that in subgroup analyses stratified by gender, ethnicity, age group, and BMI, our results indicated that the negative association between BUN and SII was independently significant among US adolescents with a BMI ≥ 25.5 kg/m 2 [− 4.77 (− 8.24, − 1.30)], but was not statistically significant in all models with a BMI < 25.5 kg/m 2 .

We performed smoothed curve fitting (Fig.  2 ) further in order to characterize the inverse relationship between BUN and SII. Using a two-stage linear regression model (Table 4 ), we found that stratified analyses by BMI revealed an inverse U-shaped curve for American adolescents with BMI < 25.5 kg/m 2 and BMI ≥ 25.5 kg/m 2 (Fig.  3 ), with an inflection point of 13 (mg/dL).

Smooth curve fitting.

BMI Smooth curve fitting.

The central finding of this study is that there is a significant negative correlation between SII and BUN levels in the US adolescent population. This means that as SII increases, indicating increased inflammation and immune activity, BUN levels show a downward trend.This result not only reveals the potential importance of SII and BUN levels in adolescent health monitoring, but also highlights the complex interactions between the immune system and metabolic processes during adolescence.

First, elevated SII usually reflects the body's response to inflammation 33 , 34 , 35 , 36 . In adolescence, this response may be more complex, as the immune system is rapidly developing and adjusting 37 , 38 , 39 . For example, one study demonstrated differences in immune cell ratios and activity in adolescence compared to adults, which may influence levels of inflammatory markers 40 . Second, the decrease in BUN levels may reflect specific changes in body and kidney function in inflammatory states 41 , 42 , and in certain pathological states, such as severe inflammation or infection, the body may respond to external stresses by altering metabolic pathways, such as by increasing protein synthesis and decreasing catabolism, which may similarly affect BUN levels. Also, even in the presence of impaired renal function, the kidneys may partially compensate for decreased excretion of urea by increasing the excretion of other metabolic wastes (e.g., creatinine). In addition, the kidneys may regulate BUN levels by altering urea reabsorption, especially in mild or moderate renal insufficiency. This is particularly critical in the adolescent population, as they are at a critical stage of growth and development. The kidneys, as the main organs of metabolism and excretion, are particularly sensitive to hormonal changes 43 . It has been suggested that hormonal fluctuations during adolescence may affect renal function and regulation 44 . During the rapid growth phase of adolescence, the metabolic activity of an organism tends to support the establishment and development of new tissues, which typically requires more anabolic processes to meet the demands of growth. As a result, the breakdown of proteins and other key nutrients may be reduced, which helps to explain why BUN levels may be lower during this growth phase, as the primary source of urea is the breakdown of proteins. Notably, this study found that the relationship between SII and BUN remained significant even after controlling for age, gender, race, and BMI. This finding suggests that the association between SII and BUN may be driven by deeper physiological processes 8 . For example, gender differences may influence immune responses and metabolic processes in adolescents 45 , 46 , 47 . One study showed significant differences in immune response between adolescent females and males, which may influence their inflammatory markers 48 . In addition, the influence of lifestyle factors, such as diet and exercise, on SII and BUN cannot be ignored. In adolescents, the occurrence of dehydration may be closely related to lifestyle (e.g., physical activity, drinking habits, etc.) due to physiological and behavioral peculiarities. Dehydration leads to a decrease in body water, which increases blood viscosity and plasma osmolality. This change stimulates the kidneys to conserve water by decreasing the volume of urine, which in turn leads to higher blood urea nitrogen (BUN) concentrations. In adolescence, when renal function and metabolic status may be more active or unstable due to growth and development, dehydration may lead to more pronounced changes in these indicators. At the same time, dehydration may affect the counts and ratios of these cells, including platelets, neutrophils, and lymphocytes, through several mechanisms. First, hemoconcentration due to dehydration can artificially increase cell counts in the blood, including neutrophils and platelets, which may lead to elevated SII. Second, prolonged or severe dehydration may affect the body's immune response by affecting the cellular function and survival environment, which in turn may affect the body's immune response 49 , 50 .

Several studies have noted that SII tends to be elevated and positively correlated with BUN levels in chronic disease states such as chronic kidney disease or cardiovascular disease 24 , 51 , whereas our study observed a negative correlation between SII and BUN in a generally healthy group of adolescents. This difference may stem from the different health states of the sample groups. In chronic disease states, the inflammatory response may be more pronounced, thus affecting renal function and leading to elevated BUN. In healthy adolescents, on the other hand, lower SII may reflect lower levels of inflammation and normal renal function, so BUN is maintained at a lower level. Secondly, some studies have found that SII and BUN may be positively correlated in metabolic abnormalities conditions such as metabolic syndrome 52 , which again differs from our findings. This may be due to the fact that inflammation and metabolic abnormalities in patients with metabolic syndrome may lead to impaired renal function, which in turn affects BUN levels. In the general population of adolescents, this relationship is less pronounced due to the lack of significant metabolic disease burden.

Overall, this study emphasizes the importance of focusing on SII and BUN levels in adolescent health surveillance. It is hoped that it will provide new perspectives on adolescent health management and disease prevention, and provide a more solid foundation for further subsequent studies. Of course there are some limitations and shortcomings in our study. First, our data were obtained from the NHANES database, and although this is a nationally representative health and nutrition survey data, its data quality and reliability still need to be further verified. Second, because of the cross-sectional design of this study, we were only able to observe a correlation between SII and BUN, and could not determine whether, and what kind of causal relationship existed between them. Future studies should adopt a longitudinal study design to track individual changes over time in order to reveal causality more accurately. Finally, our sample may not have fully covered adolescents of different ethnic and socioeconomic backgrounds, which may affect the generalizability of the findings. Future studies should include a wider range of population groups to ensure broad applicability and validity of the results. Finally, our research model did not encompass the effects of factors such as diet, exercise, and sleep on SII and BUN, and future studies should include detailed information on these variables to more fully understand their relationship with SII and BUN.

In response to the limitations mentioned above, for future studies, we consider that improvements can be made in the following ways: First, in order to determine if, and what kind of causal relationship exists between SII and BUN, future studies should utilize a longitudinal research design. Second, expanding the diversity of the sample to include adolescents from different ethnic and socioeconomic backgrounds will ensure broader representation and applicability of the findings. This can be accomplished by collaborating multiple different districts for multi-center sample collection. Further, laboratory studies can be conducted to explore the interactions between immune response, metabolic regulation, and hormonal regulation at the molecular level. This can be accomplished through biochemical analysis of biological samples such as blood and urine. Finally, adolescent-specific intervention studies, such as diet and exercise programs, could be designed and implemented to assess how these interventions affect SII and BUN levels, thereby providing more specific recommendations for health improvement. With these improvements, future studies will not only overcome the limitations of existing studies, but also provide a more in-depth and comprehensive scientific basis for adolescent health surveillance and disease prevention.

Overall, this study revealed a significant negative correlation between BUN levels and SII in a population of US adolescents and found that this association showed variability among adolescents with different BMIs. This finding has key implications for a deeper understanding of metabolic and immune functions during adolescence and provides new research directions for future studies in related areas. Future studies should aim to investigate the biological mechanisms behind this relationship and its possible impact on adolescent health.

Data availability

Publicly available datasets were analyzed in this study. This data can be found here: www.cdc.gov/nchs/nhanes/ .

Rogol, A. D. Sex steroids, growth hormone, leptin and the pubertal growth spurt. Endocrinol. Dev. 17 , 77–85 (2009).

Article   Google Scholar  

Ucciferri, C. C. & Dunn, S. E. Effect of puberty on the immune system: Relevance to multiple sclerosis. Front. Pediatr. 10 , 1059083 (2022).

Article   PubMed   PubMed Central   Google Scholar  

Casazza, K., Hanks, L. J. & Alvarez, J. A. Role of various cytokines and growth factors in pubertal development. Med. Sport Sci. 55 , 14–31 (2010).

Article   CAS   PubMed   Google Scholar  

Arain, M. et al. Maturation of the adolescent brain. Neuropsychiatr. Dis. Treat. 9 , 449–461 (2013).

PubMed   PubMed Central   Google Scholar  

Marsland, P. et al. Neuroendocrine and neuroimmune responses in male and female rats: Evidence for functional immaturity of the neuroimmune system during early adolescence. Eur. J. Neurosci. 55 (9–10), 2311–2325 (2021).

Salazar, J. Overview of urea and creatinine. Lab Med. 45 (1), e19–e20 (2014).

Swan, S. K. The search continues–an ideal marker of GFR. Clin. Chem. 43 (6 Pt 1), 913–914 (1997).

Arora, S. et al. Kidney function in minority children and adolescents with metabolically healthy and unhealthy obesity. Clin. Obes. 10 (1), e12345 (2019).

Article   PubMed   Google Scholar  

Chen, L. et al. Systemic immune-inflammation index as a useful prognostic indicator predicts survival in patients with advanced gastric cancer treated with neoadjuvant chemotherapy. Cancer Manag. Res. 9 , 849–867 (2017).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Ozarslan Ozcan, D. et al. Increased systemic immune-inflammation index levels in patients with dry eye disease. Ocul. Immunol. Inflam. 30 (3), 588–592 (2020).

Hirahara, N. et al. Systemic immune-inflammation index predicts overall survival in patients with gastric cancer: A propensity score-matched analysis. J. Gastrointest. Surg. 25 (5), 1124–1133 (2020).

Hu, B. et al. Systemic immune-inflammation index predicts prognosis of patients after curative resection for hepatocellular carcinoma. Clin. Cancer Res. 20 (23), 6212–6222 (2014).

Huang, H. et al. Prognostic value of preoperative systemic immune-inflammation index in patients with cervical cancer. Sci. Rep. 9 (1), 3284 (2019).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Das, J. K. et al. Nutrition in adolescents: Physiology, metabolism, and nutritional needs. Ann. N. Y. Acad. Sci. 1393 (1), 21–33. https://doi.org/10.1111/nyas.13330 (2017).

Article   ADS   PubMed   Google Scholar  

Alderman, E. M. & Breuner, C. C. Unique needs of the adolescent. Pediatrics 144 (6), 3150 (2019).

Li, X. et al. Systemic immune-inflammation index is a promising non-invasive biomarker for predicting the survival of urinary system cancers: A systematic review and meta-analysis. Ann. Med. 53 (1), 1827–1838 (2021).

Wang, Q. et al. Prognostic value of systemic immune-inflammation index in patients with urinary system cancers: A meta-analysis. Eur. Rev. Med. Pharmacol. 25 (3), 1302–1310 (2021).

CAS   Google Scholar  

Charlton, J. R. et al. Evolution of the urinary proteome during human renal development and maturation: Variations with gestational and postnatal age. Pediatr. Res. 72 (2), 179–185 (2012).

Tiwari, R. et al. Metabolic syndrome in obese adolescents is associated with risk for nephrolithiasis. J. Pediatr. 160 (4), 615-620.e2 (2011).

Zhang, Y. et al. Prognostic value of pretreatment systemic immune-inflammation index in patients with gastrointestinal cancers. J. Cell Physiol. 234 (5), 5555–5563 (2018).

Chen, X. et al. Association of systemic immunity-inflammation index with metabolic syndrome in U.S. adult: A cross-sectional study. BMC Geriatr. 21 (1), 48 (2021).

Google Scholar  

Gencer, S. et al. The relationship between systemic immune inflammation index and survival in patients with metastatic renal cell carcinoma treated with tyrosine kinase inhibitors. Sci. Rep. 11 (1), 1154 (2021).

Wang, H. et al. Association between systemic immune-inflammation index and insulin resistance and mortality. Sci. Rep. 10 (1), 15074 (2020).

Wang, H. et al. Systemic immune-inflammation index (SII) and the risk of all-cause, cardiovascular, and cardio-cerebrovascular mortality in the general population. Eur. J. Med. Res. 28 , 575 (2023).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Curtin, L. R. et al. The national health and nutrition examination survey: Sample design, 1999–2006. Vital Health Stat. 2 , 1–39 (2012).

Mmari, K. & Sabherwal, S. A review of risk and protective factors for adolescent sexual and reproductive health in developing countries: An update. J. Adolesc. Health 53 (5), 562–572 (2013).

Silva, C. A. et al. Pediatric rheumatic disease patients: Time to extend the age limit of adolescence?. Adv. Rheumatol. 58 (1), 30 (2018).

de Lijster, J. M. et al. Social and academic functioning in adolescents with anxiety disorders: A systematic review. J. Affect. Disord. 230 , 108–117 (2018).

Samal, S. & Vishalakshi, A. Abnormal uterine bleeding in adolescence. Int. J. Reprod. Contracept. Obstet. Gynecol. 7 (4), 1296 (2018).

Döring, N. & Walter, R. Alcohol portrayals on social media. DOC (2022).

Kramer, T. & Garralda, M. E. Psychiatric disorders in adolescents in primary care. Br. J. Psychiatry 173 , 508–513 (1998).

Hu, B. et al. Systemic immune-inflammation index predicts prognosis of patients after curative resection for hepatocellular carcinoma. Clin. Cancer Res. 20 , 6212–6222 (2014).

Ye, C. et al. Association between systemic immune-inflammation index and chronic obstructive pulmonary disease: A population-based study. BMC Pulm. Med. 23 (1), 295 (2023).

Biswas, T. et al. Using the systemic immune-inflammation index (SII) as a mid-treatment marker for survival among patients with stage-III locally advanced non-small cell lung cancer (NSCLC). Int. J. Environ. Res. Public Health 17 (21), 7995 (2020).

Zhai, Y. et al. Increased systemic immune-inflammation index is associated with delayed cerebral ischemia in patients with aneurysmal subarachnoid hemorrhage. Front. Neurol. 10 , 1235 (2019).

Liu, Y. et al. Association between systemic immune-inflammation index and metabolic syndrome and its components: Results from the National Health and Nutrition Examination Survey 2011–2016. J. Transl. Med. 17 (1), 347 (2019).

Lloyd, C. M. & Saglani, S. Development of allergic immunity in early life. Immunol. Rev. 278 (1), 101–115 (2017).

Simon, A. K., Hollander, G. A. & McMichael, A. Evolution of the immune system in humans from infancy to old age. Proc. R. Soc. B 2015 (282), 20143085 (1821).

Boehm, T. & Swann, J. B. Thymus involution and regeneration: Two sides of the same coin?. Nat. Rev. Immunol. 13 (11), 831–838 (2013).

Waubant, E. et al. Effect of puberty on the immune system: Relevance to multiple sclerosis. Front. Immunol. 10 , 2125 (2019).

Lee, J. et al. Genome-wide association analysis identifies multiple loci associated with kidney disease-related traits in Korean populations. PLoS ONE 13 (3), e0194044 (2018).

Prabhu, S. N. et al. Interdigital sensing system for kidney health monitoring. In Interdigital Sensors: Progress over the Last Two Decades 267–309 (Springer, 2021).

Chapter   Google Scholar  

Lohr, J. W., Willsky, G. R. & Acara, M. A. Renal drug metabolism. Pharmacol. Rev. 50 (1), 107–142 (1998).

CAS   PubMed   Google Scholar  

Ardissino, G. et al. Puberty is associated with increased deterioration of renal function in patients with CKD: Data from the ItalKid Project. Arch. Dis. Child. 97 (10), 885–888 (2012).

Peckham, H. et al. Gender-diverse inclusion in immunological research: Benefits to science and health. Front. Med. 9 , 909789 (2022).

Manuel, R. S. J. & Liang, Y. Sexual dimorphism in immunometabolism and autoimmunity: Impact on personalized medicine. Autoimmun. Rev. 20 (4), 102775 (2021).

Ngo, S. T., Steyn, F. J. & McCombe, P. A. Gender differences in autoimmune disease. Front. Neuroendocrinol. 35 (3), 347–369 (2014).

Klein, S. L. Immune cells have sex and so should journal articles. Endocrinology 153 (6), 2544–2550 (2012).

Dill, D. B. & Costill, D. L. Calculation of percentage changes in volumes of blood, plasma, and red cells in dehydration. J. Appl. Physiol. 37 (2), 247–248. https://doi.org/10.1152/jappl.1974.37.2.247 (1974).

Mitchell, J. B. et al. Effect of exercise, heat stress, and hydration on immune cell number and function. Med. Sci. Sports Exerc. 34 (12), 1941–1950 (2002).

Shi, L. et al. Association between systemic immune-inflammation index and low muscle mass in US adults: A cross-sectional study. BMC Public Health 23 (1), 1416 (2023).

Zeng, P. et al. Association of systemic immunity-inflammation index with metabolic syndrome in U.S. adult: A cross-sectional study. BMC Geriatr. 24 (1), 61 (2024).

Download references

Author information

These authors contributed equally: Cheng Guo and Qinhui Cai.

Authors and Affiliations

Comprehensive Pediatrics & Pulmonary and Critical Care Medicine, Kunming Children’s Hospital, No.28, Shulin Street, Kunming, 650103, Yunnan Province, China

Cheng Guo, Yang Li, Feng Li & Kai Liu

Pediatric Department, Qionghai People’s Hospital, No.33, Fuhai Road, Qionghai, 571400, Hainan Province, China

You can also search for this author in PubMed   Google Scholar

Contributions

CG: data analysis, software, and writing—original draft. FL: formal analysis and writing—original draft. YL: methodology and software. QHC: data analysis. KL: conceptualization, and writing—reviewing and editing. All authors contributed to the article and approved the submitted version.

Corresponding author

Correspondence to Kai Liu .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Guo, C., Cai, Q., Li, Y. et al. A cross-sectional National Health and Nutrition Examination survey-based study of the association between systemic immune-inflammation index and blood urea nitrogen levels in United States adolescents. Sci Rep 14 , 13248 (2024). https://doi.org/10.1038/s41598-024-64073-w

Download citation

Received : 03 February 2024

Accepted : 05 June 2024

Published : 10 June 2024

DOI : https://doi.org/10.1038/s41598-024-64073-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Cross-sectional study
• Adolescents
• Blood urea nitrogen
• Systemic Immune Inflammatory Index

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Advertisement

Developing a prognostic model for skin melanoma based on the persistent tumor mutation burden and determining IL17REL as a therapeutic target

• Open access
• Published: 20 June 2024
• Volume 150 , article number  313 , ( 2024 )

Cite this article

You have full access to this open access article

• Mingze Xu 1   na1 ,
• Xinyi Ma 1   na1 ,
• Yuchong Wang 1   na1 ,
• Ziqin Yu 2 ,
• Xiaoli Zheng 3 ,
• Haiying Dai 1 &
• Chunyu Xue 1  

One popular and well-established marker for the immune checkpoint blockade (ICB) response is tumor mutation burden (TMB). Persistent TMB (pTMB), a subset of TMB, provides a better indicator to predict patient ICB therapy outcomes, as shown by some studies. Immune checkpoint drugs have significantly changed how melanoma is treated in recent years.

In this study, we integrated the TCGA-SKCM database and data of pTMB of TCGA from the paper that first mentioned pTMB and analyzed mutational and Immune characteristics associated with pTMB level in SKCM. Next, the predictive DEGs were identified the subgroups of pTMB by Cox regression and LASSO analyses to construct a pTMB-related signature. Finally, the expression and Biological functions of signature genes was detected, and further validated in vitro assay.

In the current research, we explored the mutational and immunological features related to the level of TMB in cutaneous melanoma (CM). The high-pTMB subgroup exhibited an increasing incidence of gene changes and higher levels of immune cell infiltration. Subsequently, we established a pTMB-related signature based on the predictive DEGs and found the biological features and immune-associated variables between two distinct risk groups. Lastly, the results of the clinical sample validation demonstrated that the expression of IL17REL was down-regulated in the collected samples of individuals with CM. The in vitro assay results indicated that IL17REL effectively suppressed the proliferation, clonality, and migration of CM cells.

In conclusion, we have developed a prediction model associated with TMB and subsequently validated the potential influence of IL17REL on Overall Survival (OS) in patients diagnosed with melanoma.

Similar content being viewed by others

RPTOR mutation: a novel predictor of efficacious immunotherapy in melanoma

Predicting response to checkpoint inhibitors in melanoma beyond pd-l1 and mutational burden.

PANoptosis-related prognostic signature predicts overall survival of cutaneous melanoma and provides insights into immune infiltration landscape

Avoid common mistakes on your manuscript.

Skin cancer developing from melanocyte stem cells and fully differentiated melanocytes is known as cutaneous melanoma (CM), a highly aggressive dermal carcinoma (Centeno et al. 2023 ). Melanomas, including around 1 in 5 skin cancers, are estimated to have affected approximately 325,000 individuals worldwide in 2020. Skin cancers, the most frequently diagnosed category globally, are anticipated to be responsible for over 1.5 million new cases in 2020 (Arnold et al. 2022 ). In recent years, immune checkpoint inhibitors have brought about a significant paradigm shift in treating melanoma, particularly in cases of advanced melanoma (Serratì et al. 2022 ). The accepted standard adjuvant therapy for managing and treating CM (stage III or IV) is using an inhibitor of the programmed cell death protein − 1 (PD-1) (Patrinely et al. 2021 ; Carlino et al. 2021 ). Although better results have been linked to immune checkpoint blockade (ICB), roughly half of patients do not see long-term benefits (Jie et al. 2022 ). Various biomarkers, such as tumor neoantigen burden (TNB) (Luo et al. 2022 ) and tumor mutation burden (TMB) (Mcgrail et al. 2021 ), have been reported for utilization in predicting ICB response; however, the findings of these indicators do not reliably predict the clinical outcome of patients. Building innovative and reliable prediction technologies and tools is necessary for precise individual assessment and pre-selection of suitable therapies for patients.

The primary biomarker for identifying cancer patients who can benefit therapeutically from ICB is the high TMB (Jung et al. 2023 ). Within the tumor mutation burden framework, all mutations are considered to be of similar significance, with variations observed solely in terms of mutation quantity. In the context of immunogenicity, specific mutations exert greater influence than others (Leung and Mcgranahan 2023 ). Hence, it is not always feasible for TMB to consistently demonstrate clinical efficacy in predicting the response to cancer immunotherapy. The concept of persistent TMB (pTMB), which denotes mutations that always elicit immune tumor control throughout the progression of tumors, was initially introduced by Niknafs et al. ( 2023 ). Significantly, the study's authors highlighted that pTMB has superior predictive capabilities for tumor ICB response compared to TMB. This finding offers novel perspectives for the precise prognosis of patients with CM.

This study examines the pTMB features of melanoma patients by analyzing the data from the Gene Expression Database (GEO), the Cancer Genome Atlas Program (TCGA), and all relevant scientific data from research by Niknafs et al. We developed a signature for predicting melanoma prognosis and response to chemotherapy and immunotherapy using Cox-Lasso regression based on the discovered differential genes with prognostic importance, allowing for individualized patient treatment regimens.

Materials and methods

Samples of melanoma patients.

The collection of clinical samples from the patients conformed to the requirements stated in the Declaration of Helsinki. Before donating tumor tissue, all patients provided their informed consent by signing the necessary documentation. The surgeries were performed based on clinical indications; only residual tumor material was contributed to the research. The sample of patients consisted of three individuals. Pathological biopsies were conducted to diagnose two cases.

Furthermore, extensive resection of the primary tumor was performed. The other patients were diagnosed during the surgery via frozen section pathology. A pathologist sectioned the surgically excised tumor tissues in the operation room. The western blotting assay was performed, for which the tumor and the normal tissue portions were collected and cryopreserved from the patients.

Data collection and processing

The gene expression profile of TCGA- SKCM (log 2 (FPKM + 1) conversion) was downloaded from the R package "TCGA biolinks". The TCGA official post-correction survival information (OS) and clinical data (including gender, age, grade, stage, etc.) were collected from a published work by Liu et al. ( 2018 ). The data of pTMB of TCGA was obtained from (Niknafs et al. 2023 ), which included 107 patients of SKCM (skin cutaneous melanoma). Moreover, the GSE65904 expression data set for model validation was retrieved from the GEO database ( https://www.ncbi ). Exclusion criteria were used for patients lacking survival information or had insufficient clinical data.

Mutational characteristics associated with pTMB level in SKCM

The pTMB was divided into two subgroups, high-pTMB(H-pTMB) and low-pTMB(L-pTMB), and was defined by an optimal cutoff value. Subsequently, the Kaplan–Meier (KM) survival analysis was used for the prognosis of the two subgroups. Using the R "maftools" package, the waterfall plot of somatic mutations in the pTMB subtypes was generated. Additionally, the R "ggplot2" package was employed to create a correlation dot plot, illustrating the associations between pTMB and several factors, including TMB, TNB, homologous recombination deficiency (HRD), and chromosomal instability (CIN). The TMB, HRD, and TNB data for cutaneous melanoma were obtained from a study by Thorsson et al. ( 2018 ), and the CIN statistics were retrieved from the research done by Drews et al. ( 2022 ). Further examination was conducted to assess the correlation between various subgroups of pTMB and the clinicopathological characteristics of cutaneous melanoma.

Immune characteristics associated with pTMB level in SKCM

A comprehensive investigation was conducted to assess the relationship between various pTMB groups and immunological status in SKCM patients. The calculations on the ESTIMATEScore, ImmuneScore, TumorPurity, and StromalScore of both the low- and high-pTMB subgroups were done using the R package "ESTIMATE" (estimation of stromal and immune cells in malignant tumor tissues using expression data). For the quantification of the relative frequencies of the types of cells that had infiltrated the tumor-immune microenvironment (TIME), a ssGSEA (single-sample gene set enrichment analysis) method was used. The gene set of the infiltrating immune cell types identified in each TIME was acquired from a study by Charoentong et al. ( 2017 ). The R package "GSVA" (gene set variation analysis) was used to compare immune pathways at various pTMB levels. To generate heat maps, the R package "ComplexHeatmap" was used.

Establishing and validating the pTMB-related signature

The DEGs (differentially expressed genes) were identified between the subgroups of pTMB. The limma algorithm was used ( p  < 0.05). Subsequently, a univariate Cox regression analysis was conducted to determine the predictive DEGs. Using the "glmnet" package in R software, regression analysis was done by LASSO (Least Absolute Shrinkage and Selection Operator). This identified the signature genes and eliminated the overfitting issue. Patients' risk scores were determined by evaluating the level of expression for each prognostic gene with its related coefficient of regression.

where exp i represents the gene expression level, β i indicates the estimated regression coefficient value, and n represents the number of signature genes.

Patients diagnosed with SKCM were classified into two categories, namely high-risk and low-risk groups. This was based on the respective median risk scores. Subsequently, the survminer (survival analysis and visualization) R package was used to evaluate the OS of high- and low-risk categories of SKCM-classified patients. The curve of time-dependent ROC (receiver operating characteristic) was determined using the R "survminer" and "timeROC" utilities. In addition, univariate and multivariate Cox analyses were conducted to determine the predictive risk scores and independent prognostic values. Moreover, the formula that calculated the risk score for cohort validation was also used here. Afterward, the multivariate and univariate Cox analysis assessed the risk score independence as a prognostic determinant for patients afflicted with cutaneous malignant melanoma.

Differences in biological characteristics between the prognostic signature low- and high-risk groups

Based on a p- value cutoff of < 0.05, the prognostic markers of high-risk and low-risk groups' distinct pathways were evaluated, which was analyzed using the R "GSVA" package. Moreover, the heat maps were generated using the R "ComplexHeatmap" package.

Establishment of a nomogram model and clinical correlation analysis

Based on age, sex, clinical stage, Breslow depth, and pTMB-related features, a nomogram was constructed. To increase the clinical validation value even further, the actual and expected probabilities of 1, 3, and 5-year OS have been determined using calibration curves. The discriminatory capacity of each component to SKCM was analyzed using the ROC.

To analyze the differential expression among the tumor and normal samples in SKCM, the association between RiskScore and clinical characteristics was examined. Additionally, signature genes' predictive value and clinical importance were assessed to validate and identify potential candidate genes.

Immune-associated characteristic differences and assessment of the drug sensitivity

The study investigated the microenvironmental variations of tumors concerning the prognostic hallmark low- and high-risk subgroups using a ssGSEA method. Gene sets specifying TIME invading immune system cells were obtained explicitly from the research conducted by Charoentong et al. ( 2017 ). The 29 gene sets representing immunological properties were taken from a study by He et al. ( 2018 ). The enrichment level of immunological characteristics between low- and high-risk subgroups of predictive signature was then qualified using the ssGSEA algorithm. Following this, a systematic search was conducted for expression profiles of ICB genes that are accessible to the public and provide comprehensive clinical data. Thus, the study included two immunotherapy cohorts: one with metastatic melanoma patients receiving anti-PD- 1 antibody treatment (referred to as Cohort PRJEB23709 (Gide et al. 2019 )) and another involving melanoma patients receiving anti-CTLA- 4 antibody intervention (referred to as Cohort phs000452.v2.p1).

SubMap compared expression profiles to determine treatment impact. Therefore, the SubMap algorithm predicted the probability of anti-PD- 1 and anti-CTLA- 4 therapy responses. The data and its associated annotations were from 47 cutaneous malignant melanoma patients from published research by Lux et al. ( 2017 ).

Western blot

Tumor cells or tissues were taken from patients and subjected to protein extract. Normal cells and tissues (non-cancerous) were also collected for comparison. Proteins were extracted by radio immunoprecipitation assay (RIPA; Shanghai Life Mode Engineering, Shanghai, China) and phenylmethylsulfonyl fluoride (PMSF; Shanghai Life Mode Engineering, Shanghai, China). The BCA protein assay kit (bicinchoninic acid; Shanghai Dongsheng Biotechnology, Shanghai, China) assessed the extracted protein concentration.

The PVDF (polyvinylidene difluoride) membrane, of 0.22 μm size (Millipore ISEQ00010, USA), was incubated overnight at 4 °C with an anti-IL17REL (1:1000, Thermo Scientific). Subsequently, the membrane was incubated with a 1:2000 dilution of the secondary antibody conjugated with horseradish peroxidase (HRP; Abcam, Cambridge, UK). The detection and visualization of the protein were carried out using the Prime Western Blotting Detection Reagent (Cytiva, UK). The ChemiDoc MP imaging system (Tanon 4800, Shanghai, China) detected chemiluminescence, and the ImageJ software was used to analyze the bands' gray values.

Real-time quantitative PCR (RT-qPCR)

Utilizing the TRIzol reagent (Invitrogen, Waltham, MA, USA), total RNA was isolated from the cells of each group. Afterward, the RNA sample was subjected to reverse transcription using the reverse transcription kit (Tiangen Biotechnology, Beijing, China). The 2 × SYBR Green qPCR Master Mix (Shanghai Dongsheng Biotechnology, Shanghai, China) was used. The internal control used in this study was β-actin . The relative expression of the gene was calculated using the 2-ΔΔCt technique. The primers utilized are given in Table  1 .

Cell culture and transfection

A375 (catalog number CL-0014) and A875 (catalog number CL-0255) were brought from Procell Life Science & Tech-nology.Co.,Ltd, and they were cultured in Dulbecco's Modified Eagle Medium (DMEM; Thermo Fisher Scientific, USA) supplemented with 10% fetal bovine serum (FBS). After adding the FBS, 100 U / mL of penicillin and 100 g / mL of streptomycin were introduced to ensure sterility. Both cell lines were incubated at standard growth conditions.

The plasmids overexpressing the IL17REL gene and their corresponding negative controls were obtained from Generay Biotech (China). Lipofectamine 2000 (Invitrogen, USA) was used for transfecting the human A375 and A875 cells.

Measurement of the proliferation of cells

The BeyoClick™ EDU-55 cell proliferation detection kit (Beyotime, China) was prepared per the manufacturer's instructions. The kit provided a 5-Ethynyl-2′-deoxyuridine (Edu) solution from which a working solution was prepared and added to cells for 2 h. The cells were then fixed using a 4% paraformaldehyde solution and eventually treated with a 0.3% Triton X-100 permeability solution in a dark environment for 30 min. Hoechst nuclear fluorescence microscopy was used to detect EdU-stained cells.

Measurement of intracellular ROS in cells

The production of intracellular reactive oxygen species (ROS) was quantified using a commercially available ROS detection kit (Beyotime, China). Concisely, 3 × 10 5 cells were cultured in a 6-well plate and incubated overnight in standard growth conditions. The cells were subjected to staining using a concentration of 10 µM of DCFH-DA at 37 °C for 30 min. The images were taken and quantified.

Colony formation assay

The A375 and A875 cell lines were seeded in a 6-well plate. Following a 14-day incubation period, the cells were treated with 100% methanol for fixation and subjected to staining with a 0.5% solution of crystal violet. Eventually, the colonies were systematically counted, and images were taken.

Transwell assay

To assess the capacity of cells for transwell invasion, a volume of 100 µL containing 5 × 10 4 cells in incomplete DMEM medium (serum-free) was introduced into transwell inserts (Corning, USA). As a nutritional attractant, 10% FBS was added to serum-free DMEM and put in the lower section of the transwell experiment. The cells on the bottom surface were preserved with 4% poly-formaldehyde (Beyotime, China) for 30 min after conducting a 16-h invasion experiment. Subsequently, for 30 min, these cells were stained with a 0.4% crystal violet solution (Beyotime, China). After removing cells from the top surface, the cells on the bottom were quantified by microscopic observation.

Wound healing

The cells were gently scraped using a pipette, following the fusing of cellular components into a 6-well plate. Photographs were captured at the time points of 0 h and 24 h after the act of scratching.

Statistical analysis

The R software (version 4.1.2) was used for statistical analysis. For the significant data analysis (like expression, infiltration ratio, and various eigenvalues, etc.), the two groups of samples were compared for differences via the Wilcoxon signed rank and compared differences between multiple groups of samples through the Kruskal–Wallis.

Using an optimal cutoff value based on pTMB enables the differential classification of groups into L-pTMB and H-pTMB categories (Supplementary Table 1). The findings demonstrated that the H-pTMB subgroup showed a significantly greater survival rate than the L-pTMB subgroup, as shown in Fig.  1 A. The proportion of gene mutations in the L-pTMB was much lower than in the H-pTMB, according to somatic mutation data of various pTMB levels (Fig.  1 B). The correlation analysis of pTMB with TMB, HRD, TNB, and CIN was also performed, and its results indicated that TMB, HRD, and TNB were positively associated with pTMB. In contrast, pTMB was negatively associated with CIN (Fig.  1 C). The distribution of clinicopathological features of pTMB and cutaneous melanoma showed significant differences between gender and pTMB (Supplementary Fig. 1A). This is unlike TMB, which was found to increase significantly with age regardless of gender in large sample data analyses (Li et al. 2022 ; Chalmers et al. 2017 ). This may indicate the advantage of pTMB in elderly CM.

Shows the mutational characteristics linked to the pTMB level in SKCM. A Prognostic values of pTMB in SKCM. B Somatic mutations in different pTMB levels. ( C ) The relationship between different mutational markers and pTMB

The results of StromalScore, ImmuneScore, ESTIMATEScore, and TumorPurity for different pTMB levels were insignificant (Supplementary Fig. 1B). Immune cell subsets were quantified using the ssGSEA method. The findings revealed that the H-pTMB group had considerably more significant levels of infiltration of immature B cells, activated CD8 + T cells, activated CD4 + T cells, and other subsets (Fig.  2 A). Upon evaluating the expression of CD274 , CTLA4 , and other immune checkpoint genes at different pTMB levels, it was seen that the H-pTMB subgroup showed a significantly higher expression of CD274 , CTLA4 , and ICOS (Fig.  2 B). The H-pTMB group may thus be more responsive to immunotherapy, according to our inference. According to the GSVA differences, the H-pTMB group had a considerably high enrichment of the regulatory pathway of autophagy (Fig.  2 C). Autophagy is directly linked to the control of the immune response in tumors. It is also essential for the proper function and survival of the immune system's effector and developmental T cells. The extracellular matrix (ECM)-receptor interaction pathway, linked to tumor resistance, was significantly higher in the L- pTMB group.

Immunological characteristics associated with pTMB concentration in SKCM. A ssGSEA immune infiltration with different pTMB levels. B Immune checkpoints with different pTMB levels. C GSVA of the immune pathway with varying levels of pTMB

Identification of pTMB-related signature

R software's "limma" package detected DEGs among the low-pTMB and high-pTMB subgroups. A comprehensive analysis revealed that a total of 2292 genes had differential expression patterns, with 1490 genes being up-regulated and 802 genes being down-regulated (Supplementary Table 2).

The univariate Cox regression analysis based on pTMB characteristics found 972 prognostic genes ( p  < 0.05). Following the intersection with differential genes, 217 genes having prognostic significance were found. The forest plot was constructed using the top 20 genes with the lowest p -values (Fig.  3 A). Due to the extensive number of genes, which poses challenges for clinical identification, the LASSO regression model to refine the focus and determine the trajectory of the independent variables under study (Fig.  3 B). As lambda increased, independent variable coefficients gradually decreased to zero. The RiskScore, which is the gene-based survival risk score model, was constructed using 7 LASSO-coefficient-carrying genes, including IL17REL , SDC3 , RHOBTB2 , GSTA4 , MDFI , PTK7 , and FGF18 , based on the lambda value using LASSO. The confidence interval associated with each lambda value proves that the model achieves optimality when the number of genes is 7. This further supports the reliability of the candidate gene selection process. The RiskScore models were constructed using a ten-fold cross-validation approach, wherein the coefficients and expression levels of 7 specific genes were utilized to evaluate their influence on the OS outcome.

Establishing and validating the signature associated with pTMB. A Top 20 DEGs with prognostic values. B LASSO COX regression(with optimal lambda) identifying 7 host genes. C Patient statuses and expression patterns for seven host genes in the training cohort's high- and low-risk categories. The K-M survival curve and the AUC curve have distinct outcomes. D Status and expression patterns of seven host genes in the testing cohort's high- and low-risk patient groups. K-M survival curve and ROC curve demonstrating dissimilar outcomes

Based on median RiskScore, TCGA- SKCM cohort samples were divided into low- and high-risk categories (Supplementary Table 3). The KM survival analysis indicates that high-risk patients had a significantly lower OS than low-risk individuals. Additionally, the TCGA-SKCM cohort's OS could be predicted by the RiskScore, AUCs (area under the curve) for 1, 2, and 3 years were 0.819, 0.857, and 0.816, correspondingly (Fig.  3 C).

Using the same technique as the validation set GSE65904, the RiskScore model created using the TCGA-SKCM cohort was assessed for stability. The results demonstrated that low-risk SKCM had more substantial survival benefits, consistent with the TCGA-SKCM cohort (Fig.  3 C).

The univariate Cox regression analysis shows a strong association between TCGA-SKCM cohort risk score and OS (TCGA-SKCM cohort: HR = 5.99, 95% CI = 2.42–14.80, p  < 0.001; GSE65904 cohort: HR = 1.67, 95% CI = 1.13–2.48, p  = 0.01). Multivariate Cox regression analysis showed that risk score constituted an independent OS predictor (TCGA-SKCM cohort: HR = 9.72, 95% CI = 3.08–30.71, p  < 0.001; GSE65904 cohort: HR = 1.63, 95% CI = 1.10–2.42, p  = 0.02) (Fig.  4 A). A detailed investigation of the clinical applicability of the risk score signature. The present study observed that no clinical indicators exhibited positive findings in the independent prognostic analysis. Consequently, all the clinical indicators and risk scores were utilized to construct nomogram models and generate a calibration curve. The results are depicted in Fig.  4 B.

Development of the nomogram model. A Univariate and B multivariate Cox regression analyses for the risk score and other clinical variables of the training cohort. C Univariate and D multivariate Cox regression analysis for the risk score and other clinical characteristics of the testing cohort. E The construction of the nomogram aimed to develop a predictive model for estimating the probabilities of survival at 1-, 3-, and 5-year intervals. F Correction curve showing the consistency between predicted survival possibilities and observed survival rate

The biological traits of low- and high-risk subgroups were assessed using GSVA.

The enrichment proportion of basal cell carcinoma patients in the high-risk group was significantly higher than in the low-risk group.

The enrichment fraction was significantly higher in the high-risk subgroup than in the low-risk subgroup in basal cell carcinoma, melanogenesis, and other pathways (Fig.  5 ).

Differences in the biological features of the pTMB-related signature

Immune-associated characteristic differences and prediction of potential drug therapy between the prognostic signature low- and high-risk groups

Several immune cell subsets were quantified using ssGSEA, showing that high-risk subgroups had more CD56 bright natural killer cell infiltration. In contrast, the activated CD4 + T cells and eosinophils invaded the low-risk subgroup (Fig.  6 A). The difference in the immune function characteristics revealed that the levels of mast cells were considerably elevated in the high-risk subgroup. On the other hand, the low-risk subgroup had more NK cells, T cell co-inhibition, and Th1 cells (Fig.  6 B). Checkpoint expression analysis showed a statistically significant increase of CTLA4 in the low-risk subgroup (Fig.  6 C).

Variations in immune-related characteristics and possible drug treatment prediction in high-risk and low-risk populations. A Immune cell infiltration in high-risk and low-risk groups. B The graph displays the variations in immune function features between the low-risk and high-risk cohorts. C Boxplot showing immune checkpoint expression differences between low- and high-risk SKCM patients. D Drug sensitivity in low- and high-risk SKCM, including CTLA4 and PD-1. E Heat map for the response possibility of anti-PD-1 and anti-CTLA-4 treatment in the two risk groups

The melanoma treatment-associated sensitivity analysis conducted on the low- and high-risk subgroups indicates that the low-risk subgroup exhibited more positive responses to CTLA4 and PD-1 therapy than the high-risk subgroup (Fig.  6 D). Figure  6 E shows that the Submap approach compared the immunotherapy efficacy. The findings indicated a substantial similarity between the low-risk subgroup and the data from PD-1 treatment, implying that this group exhibited sensitivity to immunotherapy.

Expression of signature genes in SKCM

The differentially expressed genes were evaluated in normal and tumor samples to validate the risk-scoring model. For this, seven potential genes were investigated. The findings showed that in SKCM, the expression of FGF18 , MDFI , GSTA4 , and IL17REL were lowered. In contrast, SKCM increased the expression levels of the remaining two genes, except for PTK7 , which did not show a statistically significant difference (Fig.  7 A). The prognostic value of 6 genes in SKCM was analyzed and revealed that just the IL17REL gene exhibited expression levels in SKCM that correlated with OS. Specifically, it was shown that IL17REL expression was relatively low in SKCM cases, indicating a poor prognosis (Fig.  7 B). Therefore, validation of a prognostic signature by examining the effect of the IL17REL gene in tumor tissues and cell lines was done.

Expression of signature genes in SKCM. A Differentially expressed genes were evaluated on samples obtained from both normal and malignant tissues. B The K-M survival curve was produced to analyze the 7 host genes

Biological functions of the selected gene

IL17REL expression was detected in 3 in situ collected CM tissues, and the results from western blot (WB) showed a decrease in IL17REL gene expression, consistent with the bioinformatics analysis (Fig.  8 A). To further verify the effect of the IL17REL gene on the prognosis of CM, the overexpression efficiency of the IL17REL gene in human A375 and A875 cells was conducted using qRT-PCR and WB (Figure), and OE—IL17REL-1 and OE—IL17REL-2 were selected for further investigations (Fig.  8 B). The findings from the EdU experiment demonstrated that the upregulation of IL17REL had a detrimental effect on the proliferative capacity of malignant melanoma (MM) cells (Fig.  8 C). The levels of ROS generation in MM cells exhibited a reduction after the overexpression of IL17REL (Fig.  8 D). In addition, it was observed that the overexpression of IL17REL resulted in the suppression of MM cell migration, as evidenced by the findings from wound healing and transwell experiment (Fig.  8 E). The A375 and A875 cells overexpressed IL17REL had significantly higher cell proliferation and colony formation than the normal control cells (Fig.  8 F).

IL17REL inhibited the proliferation, clone, and migration in vivo and in vitro. A The expression of IL17REL was down-regulated in CM samples. B IL17REL was overexpressed in A375 and A875 cells. C OE-IL17REL inhibited proliferation in A375 and A875 cells.( D OE-IL17REL decreases ROS in A375 and A875 cells. E , F OE-IL17REL inhibited migration, invasion, and clone in A375 and A875 cells

Melanoma exhibits a notable degree of immunogenicity owing to its elevated load of genetic mutations and neoantigens, which can potentially trigger the commencement of the tumor cell elimination phase (Kalaora et al. 2022 ; Puig-Saus et al. 2023 ). The correlation between tumor mutation burden and the response to ICB has been widely acknowledged, making it a valuable predictive technique for assessing the outcomes of patients undergoing ICB treatment (Liu et al. 2019 ). The immunoediting theory suggests that, when subjected to immunotherapy, cancer cells acquire resistance against effector immune cells by favoring the growth of clones with reduced immunogenicity. According to a study, neoantigen loss was seen as a result of either the removal of tumor subclones or the loss of copy numbers (Łuksza et al. 2022 ). The observed loss was associated with the emergence of acquired resistance to immune checkpoint therapy (Anagnostou et al. 2017 ). In summary, these findings demonstrate the importance of accurately forecasting the clonality and heterogeneity of neoantigens.

Niknafs et al., 2023 , reported a collection of mutations characterized as "persistent" because of their reduced susceptibility to lose or develop immunoediting during tumor progression. These mutations primarily manifest as genomic and chromosomal deletions, constituting a minor proportion (10%) of somatic mutations. They defined it as pTMB, which refers to the cumulative count of single-copy and multi-copy mutations. The Whole Exome Sequencing (WES) research conducted on a collection of tumor samples both before and after ICB treatment revealed that persistent mutations exhibited a reduced tendency to induce subclonal loss during tumor progression within ICB (Davoli et al. 2017 ). Furthermore, there was no association observed between tumor clonal heterogeneity and the presence of persistent mutations. The researchers assessed the varying reclassification of cancer in 33 distinct tumor types. The results revealed that the average reclassification rate for the low/ high TMB subgroup, compared to the persistent low/high TMB subgroup, was 33%. This suggests that TMB and pTMB varied across all forms of cancer. In clinical applications, pTMB is better than TMB in predicting ICB response, and the authors further suggest that the predictive capacity of TMB to clinical outcomes primarily depends upon persistent mutations. The measurement of pTMB is an innovative method and has potential application in predicting ICB treatment outcomes in CM.

The present investigation examined the clinical and immunological associations between pTMB and CM, identified prognostically significant differentially expressed genes based on pTMB characteristics, developed a risk scoring system using pTMB-related gene modules, and investigated their prognostic utility, biological distinctions among various groups, and immune attributes for predicting potential therapeutic strategies for CM. Combined with candidate genes' differential expression and predictive value, further validation of the model in tissue specimens and in vitro experiments was done.

Among the pTMB-associated immune features, immature B cells, activated B cells, activated CD8 + T cells, and activated CD4 + T cells in the high- pTMB subgroup were up-regulated. Studies have shown that activated CD8 + and CD4 + T cells boost anti-tumor immunity. These cells are crucial in promoting the density and targeting of CD4 + /CD8 + effector T cells, enhancing immunotherapy's efficacy (Hirschhorn et al. 2023 ; Virassamy et al. 2023 ). There is a correlation between elevated levels of B-cell infiltration within the tumor microenvironment and favorable clinical outcomes in melanoma patients who undergo immunotherapy (Cabrita et al. 2020 ; Helmink et al. 2020 ). A comparison of immune checkpoint gene expression across multiple pTMB modes was performed. CD274 , CTLA4 , and ICOS exhibited a statistically significant upregulation in the high-pTMB subgroup.

Moreover, the high-pTMB subgroup showed a substantial rise in the regulatory pathway of autophagy, as seen by the GSVA results of various pTMB modes. Autophagy is essential for immune system development and affects T cell survival and function, influencing the control of immunological responses against tumors (Xia et al. 2021 ; Debnath et al. 2023 ). The ECM-receptor interaction pathway exhibited a significant enrichment in the group with low—p TMB. This enrichment was found to be connected with the development of drug resistance in tumors (Holle et al. 2016 ). Based on the study mentioned above, it is postulated that individuals with CM who exhibit high levels of pTMB may potentially be more sensitive to immunotherapy.

The current research used a risk score based on pTMB gene expression in multiple modes to construct a prognosis model and identified the differential genes with prognostic significance using analysis of different features. The results demonstrated this model's efficacy in accurately predicting patients' OS. Also, a decreased expression of the IL17REL gene was linked to worse outcomes in individuals with CM. This finding supports the validation of our proposed prognostic signature from a clinical standpoint. IL17REL encodes an IL17RE-like protein, and IL17RE is the least understood member of the IL17R family. In genome-wide association (GWAS) studies (Franke et al. 2010 ) and whole exon sequencing studies (Hu et al. 2021 ), IL17REL was found to be strongly correlated with the development of inflammatory bowel disease (IBD). Recent evidence indicates a strong association between IL17REL and the prognostic evaluation of HPV- associated head and neck squamous cell cancer (Yanan et al. 2023 ; Sun, et al. 2023 ). There needs to be more understanding regarding the expression and role of IL17REL in many cell types, particularly in tumor cells. Our study contributes novel insights to the realm of melanoma research.

In summary, this study has effectively developed and validated a predictive model linked to pTMB and has confirmed the possible impact of IL17REL on OS in individuals with melanoma. However, more experimental investigations are required to elucidate the precise molecular mechanism behind this association. Due to insufficient patient data, the study was incomplete regarding pre- and post-immunotherapy data for both the low and high-risk subgroups. Thus, the predictive models of pTMB can enhance the accuracy of patient survival predictions and serve as a valuable foundation for personalized decision-making in clinical settings.

Availability of data and materials

The data used to support the findings of this study are available from the corresponding author upon request.

Anagnostou V, Smith KN, Forde PM et al (2017) Evolution of neoantigen landscape during immune checkpoint blockade in non-small cell lung cancer. Cancer Discov 7(3):264–276

Article   CAS   PubMed   Google Scholar  

Arnold M, Singh D, Laversanne M et al (2022) Global burden of cutaneous melanoma in 2020 and projections to 2040. JAMA Dermatol 158(5):495–503

Article   PubMed   PubMed Central   Google Scholar  

Cabrita R, Lauss M, Sanna A et al (2020) Tertiary lymphoid structures improve immunotherapy and survival in melanoma. Nature 577(7791):561–565

Carlino MS, Larkin J, Long GV (2021) Immune checkpoint inhibitors in melanoma. Lancet 398(10304):1002–1014

Centeno PP, Pavet V, Marais R (2023) The journey from melanocytes to melanoma. Nat Rev Cancer 23(6):372–390

Chalmers ZR, Connelly CF, Fabrizio D et al (2017) Analysis of 100,000 human cancer genomes reveals the landscape of tumor mutational burden. Genome Med 9(1):34

Charoentong P, Finotello F, Angelova M et al (2017) Pan-cancer immunogenomic analyses reveal genotype-immunophenotype relationships and predictors of response to checkpoint blockade. Cell Rep 18(1):248–262

Davoli T, Uno H, Wooten EC et al (2017) Tumor aneuploidy correlates with markers of immune evasion and with reduced response to immunotherapy. Science 355(6322):eaaf8399

Debnath J, Gammoh N, Ryan KM (2023) Autophagy and autophagy-related pathways in cancer. Nat Rev Mol Cell Biol 24(8):560–575

Drews RM, Hernando B, Tarabichi M et al (2022) A pan-cancer compendium of chromosomal instability. Nature 606(7916):976–983

Article   CAS   PubMed   PubMed Central   Google Scholar  

Franke A, Balschun T, Sina C et al (2010) Genome-wide association study for ulcerative colitis identifies risk loci at 7q22 and 22q13 (IL17REL). Nat Genet 42(4):292–294

Gide TN, Quek C, Menzies AM et al (2019) Distinct immune cell populations define response to anti-PD-1 monotherapy and anti-PD-1/Anti-CTLA-4 combined therapy. Cancer Cell 35(2):238

He Y, Jiang Z, Chen C et al (2018) Classification of triple-negative breast cancers based on Immunogenomic profiling. J Exp Clin Cancer Res 37(1):327

Helmink BA, Reddy SM, Gao J et al (2020) B cells and tertiary lymphoid structures promote immunotherapy response. Nature 577(7791):549–555

Hirschhorn D, Budhu S, Kraehenbuehl L et al (2023) T cell immunotherapies engage neutrophils to eliminate tumor antigen escape variants. Cell 186(7):1432

Holle AW, Young JL, Spatz JP (2016) In vitro cancer cell-ECM interactions inform in vivo cancer treatment. Adv Drug Deliv Rev 97:270–279

Hu S, Vich Vila A, Gacesa R et al (2021) Whole exome sequencing analyses reveal gene-microbiota interactions in the context of IBD. Gut 70(2):285–296

CAS   PubMed   Google Scholar  

Jie X, Chen Y, Zhao Y et al (2022) Targeting KDM4C enhances CD8+ T cell mediated antitumor immunity by activating chemokine CXCL10 transcription in lung cancer. J Immunother Cancer 10(2):e003716

Jung J, Heo YJ, Park S (2023) High tumor mutational burden predicts favorable response to anti-PD-(L)1 therapy in patients with solid tumor: a real-world pan-tumor analysis. J Immunother Cancer 11(4):e006454

Kalaora S, Nagler A, Wargo JA et al (2022) Mechanisms of immune activation and regulation: lessons from melanoma. Nat Rev Cancer 22(4):195–207

Leung M, Mcgranahan N (2023) Persistence is key: refining immunotherapy response prediction. Immunity 56(3):472–474

Li CH, Haider S, Boutros PC (2022) Age influences on the molecular presentation of tumours. Nat Commun 13(1):208

Liu J, Lichtenberg T, Hoadley KA et al (2018) An integrated TCGA pan-cancer clinical data resource to drive high-quality survival outcome analytics. Cell 173(2):400–416

Liu D, Schilling B, Liu D et al (2019) Integrative molecular and clinical modeling of clinical outcomes to PD1 blockade in patients with metastatic melanoma. Nat Med 25(12):1916–1927

Łuksza M, Sethna ZM, Rojas LA et al (2022) Neoantigen quality predicts immunoediting in survivors of pancreatic cancer. Nature 606(7913):389–395

Luo K, Liu S, Shen X et al (2022) Integration of cancer stemness and neoantigen load to predict responsiveness to anti-PD1/PDL1 therapy [J]. Front Cell Dev Biol 10:1003656

McGrail DJ, Pilié PG, Rashid NU, Voorwerk L et al (2021) High tumor mutation burden fails to predict immune checkpoint blockade response across all cancer types. Ann Oncol 32(5):661–672

Niknafs N, Balan A, Cherry C et al (2023) Persistent mutation burden drives sustained anti-tumor immune responses. Nat Med 29(2):440–449

Patrinely JR, Johnson R, Lawless AR et al (2021) Chronic immune-related adverse events following adjuvant Anti-PD-1 therapy for high-risk resected melanoma. JAMA Oncol 7(5):744–748

Article   PubMed   Google Scholar  

Puig-Saus C, Sennino B, Peng S et al (2023) Neoantigen-targeted CD8+ T cell responses with PD-1 blockade therapy. Nature 615(7953):697–704

Roh W, Chen P-L, Reuben A et al (2017) Integrated molecular analysis of tumor biopsies on sequential CTLA-4 and PD-1 blockade reveals markers of response and resistance. Sci Transl Med. https://doi.org/10.1126/scitranslmed.aah3560

Serratì S, Guida M, Di Fonte R et al (2022) Circulating extracellular vesicles expressing PD1 and PD-L1 predict response and mediate resistance to checkpoint inhibitors immunotherapy in metastatic melanoma. Mol Cancer 21(1):20

Thorsson V, Gibbs DL, Brown SD et al (2018) The immune landscape of cancer. Immunity 48(4):812–830

Virassamy B, Caramia F, Savas P et al (2023) Intratumoral CD8+ T cells with a tissue-resident memory phenotype mediate local immunity and immune checkpoint responses in breast cancer. Cancer Cell 41(3):585

Xia H, Green DR, Zou W (2021) Autophagy in tumour immunity and therapy. Nat Rev Cancer 21(5):281–297

Yanan L, Hui L, Zhuo C et al (2023) Comprehensive analysis of mitophagy in HPV-related head and neck squamous cell carcinoma. Sci Rep 13(1):7480

Yuhan Sun M, Khan AAK, Mangiola S et al (2023) IL17RB and IL17REL expression are associated with improved prognosis in HPV-infected head and neck squamous cell carcinomas. Pathogens 12(4):572

Download references

This study was funded by National Natural Science Foundation of China (Grant No.81871578) and supported by Sichuan Science and Technology Program (No.2020YFSY0030).

Author information

Mingze Xu, Xinyi Ma and Yuchong Wang have contributed equally to this work as co-first authors.

Authors and Affiliations

Department of Plastic Surgery, Changhai Hospital, Naval Military Medical University, 168 Changhai Road, Shanghai, 200433, People’s Republic of China

Mingze Xu, Xinyi Ma, Yuchong Wang, Haiying Dai & Chunyu Xue

Department of Radiology, Changhai Hospital, Naval Military Medical University, Shanghai, China

Basic Medical School, Southwest Medical University, Luzhou, Sichuan, China

Xiaoli Zheng

You can also search for this author in PubMed   Google Scholar

Contributions

CX and HD designed the project and assays. MX and XM conducted the experimental assays. YW and XZ conducted the bioinformatic and statistical analysis. MX and DH collected the melanoma tissues. MX and XM wrote the paper and CX revised it. All authors have approved the final manuscript.

Corresponding authors

Correspondence to Haiying Dai or Chunyu Xue .

Ethics declarations

Conflict of interest.

None of the authors has a financial interest in any of the products or devices mentioned in this article.

Ethical approval and consent to participate

The study was approved by the Ethics Committee of the Changhai Hospital and was carried out according to the procedure of Changhai Hospital clinical research center biobank experiment.All patients who provided clinical specimens signed the written informed consent form.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

432_2024_5843_MOESM1_ESM.png

Supplementary file1 Supplementary Figure 1 A The distribution of clinicopathological features of pTMB levels and cutaneous melanoma, B The results of StromalScore,ImmuneScore, ESTIMATEScore, and TumorPurity for different pTMB levels (PNG 172 KB)

432_2024_5843_MOESM2_ESM.xlsx

Supplementary file2 Supplementary Table 1: The differential classification of groups into L-pTMB and H-pTMB categories. (XLSX 14 KB)

Supplementary file3 Supplementary Table 2: The DEGs among the low-pTMB and high-pTMB subgroups. (TXT 2447 KB)

432_2024_5843_moesm4_esm.xlsx.

Supplementary file4 Supplementary Table 3: The differential classification of groups into low- and high-risk categories. (XLSX 1857 KB)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Xu, M., Ma, X., Wang, Y. et al. Developing a prognostic model for skin melanoma based on the persistent tumor mutation burden and determining IL17REL as a therapeutic target. J Cancer Res Clin Oncol 150 , 313 (2024). https://doi.org/10.1007/s00432-024-05843-x

Download citation

Received : 23 April 2024

Accepted : 07 June 2024

Published : 20 June 2024

DOI : https://doi.org/10.1007/s00432-024-05843-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Cutaneous melanoma
• Prognostic model
• Persistent tumor mutation burden
• Therapeutic target
• Gene signature
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 14 June 2024

Associations between deep venous thrombosis and thyroid diseases: a two-sample bidirectional Mendelian randomization study

• Lifeng Zhang 1   na1 ,
• Kaibei Li 2   na1 ,
• Qifan Yang 1 ,
• Yao Lin 1 ,
• Caijuan Geng 1 ,
• Wei Huang 1 &
• Wei Zeng 1  

European Journal of Medical Research volume  29 , Article number:  327 ( 2024 ) Cite this article

199 Accesses

Metrics details

Some previous observational studies have linked deep venous thrombosis (DVT) to thyroid diseases; however, the findings were contradictory. This study aimed to investigate whether some common thyroid diseases can cause DVT using a two-sample Mendelian randomization (MR) approach.

This two-sample MR study used single nucleotide polymorphisms (SNPs) identified by the FinnGen genome-wide association studies (GWAS) to be highly associated with some common thyroid diseases, including autoimmune hyperthyroidism (962 cases and 172,976 controls), subacute thyroiditis (418 cases and 187,684 controls), hypothyroidism (26,342 cases and 59,827 controls), and malignant neoplasm of the thyroid gland (989 cases and 217,803 controls. These SNPs were used as instruments. Outcome datasets for the GWAS on DVT (6,767 cases and 330,392 controls) were selected from the UK Biobank data, which was obtained from the Integrative Epidemiology Unit (IEU) open GWAS project. The inverse variance weighted (IVW), MR-Egger and weighted median methods were used to estimate the causal association between DVT and thyroid diseases. The Cochran’s Q test was used to quantify the heterogeneity of the instrumental variables (IVs). MR Pleiotropy RESidual Sum and Outlier test (MR-PRESSO) was used to detect horizontal pleiotropy. When the causal relationship was significant, bidirectional MR analysis was performed to determine any reverse causal relationships between exposures and outcomes.

This MR study illustrated that autoimmune hyperthyroidism slightly increased the risk of DVT according to the IVW [odds ratio (OR) = 1.0009; p  = 0.024] and weighted median methods [OR = 1.001; p  = 0.028]. According to Cochran’s Q test, there was no evidence of heterogeneity in IVs. Additionally, MR-PRESSO did not detect horizontal pleiotropy ( p  = 0.972). However, no association was observed between other thyroid diseases and DVT using the IVW, weighted median, and MR-Egger regression methods.

Conclusions

This study revealed that autoimmune hyperthyroidism may cause DVT; however, more evidence and larger sample sizes are required to draw more precise conclusions.

Introduction

Deep venous thrombosis (DVT) is a common type of disease that occurs in 1–2 individuals per 1000 each year [ 1 ]. In the post-COVID-19 era, DVT showed a higher incidence rate [ 2 ]. Among hospitalized patients, the incidence rate of this disease was as high as 2.7% [ 3 ], increasing the risk of adverse events during hospitalization. According to the Registro Informatizado Enfermedad Tromboembolica (RIETE) registry, which included data from ~ 100,000 patients from 26 countries, the 30-day mortality rate was 2.6% for distal DVT and 3.3% for proximal DVT [ 4 ]. Other studies have shown that the one-year mortality rate of DVT is 19.6% [ 5 ]. DVT and pulmonary embolism (PE), collectively referred to as venous thromboembolism (VTE), constitute a major global burden of disease [ 6 ].

Thyroid diseases are common in the real world. Previous studies have focused on the relationship between DVT and thyroid diseases, including thyroid dysfunction and thyroid cancer. Some case reports [ 7 , 8 , 9 ] have demonstrated that hyperthyroidism is often associated with DVT and indicates a worse prognosis [ 10 ]. The relationship between thyroid tumors and venous thrombosis has troubled researchers for many years. In 1989, the first case of papillary thyroid carcinoma presenting with axillary vein thrombosis as the initial symptom was reported [ 11 ]. In 1995, researchers began to notice the relationship between thyroid tumors and hypercoagulability [ 12 ], laying the foundation for subsequent extensive research. However, the aforementioned observational studies had limitations, such as small sample sizes, selection bias, reverse causality, and confounding factors, which may have led to unreliable conclusions [ 13 ].

Previous studies have explored the relationship of thyroid disease and DVT and revealed that high levels of thyroid hormones may increase the risk of DVT. Hyperthyroidism promotes a procoagulant and hypofibrinolytic state by affecting the von Willebrand factor, factors VIII, IV, and X, fibrinogen, and plasminogen activator inhibitor-1 [ 14 , 15 ]. At the molecular level, researchers believe that thyroid hormones affect coagulation levels through an important nuclear thyroid hormone receptor (TR), TRβ [ 16 ], and participate in pathological coagulation through endothelial dysfunction. Thyroid hormones may have non-genetic effects on the behavior of endothelial cells [ 17 , 18 ]. In a study regarding tumor thrombosis, Lou [ 19 ] found that 303 circular RNAs were differentially expressed in DVT using microarray. Kyoto Encyclopedia of Genes and Genomes (KEGG) analysis revealed that the most significantly enriched pathways included thyroid hormone-signaling pathway and endocytosis, and also increased level of proteoglycans in cancer. This indicated that tumor cells and thyroid hormones might interact to promote thrombosis. Based on these studies, we speculated that thyroid diseases, including thyroid dysfunction and thyroid tumors, may cause DVT.

Mendelian randomization (MR) research is a causal inference technique that can be used to assess the causal relationship and reverse causation between specific exposure and outcome factors. If certain assumptions [ 20 ] are fulfilled, genetic variants can be employed as instrumental variables (IVs) to establish causal relationships. Bidirectional MR analysis can clarify the presence of reverse causal relationships [ 21 ], making the conclusions more comprehensive. Accordingly, we aimed to apply a two-sample MR strategy to investigate whether DVT is related to four thyroid diseases, including autoimmune hyperthyroidism, subacute thyroiditis, hypothyroidism, and thyroid cancer.

Study design

MR relies on single nucleotide polymorphisms (SNPs) as IVs. The IVs should fulfill the following three criteria [ 22 ]: (1) IVs should be strongly associated with exposure. (2) Genetic variants must be independent of unmeasured confounding factors that may affect the exposure–outcome association. (3) IVs are presumed to affect the outcome only through their associations with exposure (Fig.  1 ). IVs that met the above requirements were used to estimate the relationship between exposure and outcome. Our study protocol conformed to the STROBE-MR Statement [ 23 ], and all methods were performed in accordance with the relevant guidelines and regulations.

The relationship between instrumental variables, exposure, outcome, and confounding factors

Data sources and instruments

Datasets (Table  1 ) in this study were obtained from a publicly available database (the IEU open genome-wide association studies (GWAS) project [ 24 ] ( https://gwas.mrcieu.ac.uk )). There was no overlap in samples between the data sources of outcome and exposures. Using de-identified summary-level data, privacy information such as overall age and gender were hidden. Ethical approval was obtained for all original work. This study complied with the terms of use of the database.

MR analysis was performed using the R package “TwoSampleMR”. SNPs associated with each thyroid disease at the genome-wide significance threshold of p  < 5.0 × 10 –8 were selected as potential IVs. To ensure independence between the genetic variants used as IVs, the linkage disequilibrium (LD) threshold for grouping was set to r 2  < 0.001 with a window size of 10,000 kb. The SNP with the lowest p -value at each locus was retained for analyses.

Statistical analysis

Multiple MR methods were used to infer causal relationships between thyroid diseases and DVT, including the inverse variance weighted (IVW), weighted median, and MR-Egger tests, after harmonizing the SNPs across the GWASs of exposures and outcomes. The main analysis was conducted using the IVW method. Heterogeneity and pleiotropy were also performed in each MR analysis. Meanwhile, the MR-PRESSO Global test [ 25 ] was utilized to detect horizontal pleiotropy. The effect trend of SNP was observed through a scatter plot, and the forest plot was used to observe the overall effects. When a significant causal relationship was confirmed by two-sample MR analysis, bidirectional MR analysis was performed to assess reverse causal relationships by swapping exposure and outcome factors. Parameters were set the same as before. All abovementioned statistical analyses were performed using the package TwoSampleMR (version 0.5.7) in the R program (version 4.2.1).

After harmonizing the SNPs across the GWASs for exposures and outcomes, the IVW (OR = 1.0009, p  = 0.024, Table  2 ) and weighted median analyses (OR = 1.001, p  = 0.028) revealed significant causal effects between autoimmune hyperthyroidism and DVT risk. Similar results were observed using the weighted median approach Cochran’s Q test, MR-Egger intercept, and MR-PRESSO tests suggested that the results were not influenced by pleiotropy and heterogeneity (Table  2 ). However, the leave-one-out analysis revealed a significant difference after removing some SNPs (rs179247, rs6679677, rs72891915, and rs942495, p  < 0.05, Figure S2a), indicating that MR results were dependent on these SNPs (Figure S2, Table S1). No significant effects were observed in other thyroid diseases (Table  2 ). The estimated scatter plot of the association between thyroid diseases and DVT is presented in Fig.  2 , indicating a positive causal relationship between autoimmune hyperthyroidism and DVT (Fig.  2 a). The forest plots of single SNPs affecting the risk of DVT are displayed in Figure S1.

The estimated scatter plot of the association between thyroid diseases and DVT. MR-analyses are derived using IVW, MR-Egger, weighted median and mode. By fitting different models, the scatter plot showed the relationship between SNP and exposure factors, predicting the association between SNP and outcomes

Bidirectional MR analysis was performed to further determine the relationship between autoimmune hyperthyroidism and DVT. The reverse causal relationship was not observed (Table S2), which indicated that autoimmune hyperthyroidism can cause DVT from a mechanism perspective.

This study used MR to assess whether thyroid diseases affect the incidence of DVT. The results showed that autoimmune hyperthyroidism can increase the risk of DVT occurrence, but a reverse causal relationship was not observed between them using bidirectional MR analysis. However, other thyroid diseases, such as subacute thyroiditis, hypothyroidism, and thyroid cancer, did not show a similar effect.

Recently, several studies have suggested that thyroid-related diseases may be associated with the occurrence of DVT in the lower extremities, which provided etiological clues leading to the occurrence of DVT in our subsequent research. In 2006, a review mentioned the association between thyroid dysfunction and coagulation disorders [ 26 ], indicating a hypercoagulable state in patients with hyperthyroidism. In 2011, a review further suggested a clear association between hypothyroidism and bleeding tendency, while hyperthyroidism appeared to increase the risk of thrombotic events, particularly cerebral venous thrombosis [ 27 ]. A retrospective cohort study [ 28 ] supported this conclusion, but this study only observed a higher proportion of concurrent thyroid dysfunction in patients with cerebral venous thrombosis. The relationship between thyroid function and venous thromboembolism remains controversial. Krieg VJ et al. [ 29 ] found that hypothyroidism has a higher incidence rate in patients with chronic thromboembolic pulmonary hypertension and may be associated with more severe disease, which seemed to be different from previous views that hyperthyroidism may be associated with venous thrombosis. Alsaidan [ 30 ] also revealed that the risk of developing venous thrombosis was almost increased onefold for cases with a mild-to-moderate elevation of thyroid stimulating hormone and Free thyroxine 4(FT4). In contrast, it increased twofold for cases with a severe elevation of thyroid stimulating hormone and FT4. Raised thyroid hormones may increase the synthesis or secretion of coagulation factors or may decrease fibrinolysis, which may lead to the occurrence of coagulation abnormality.

Other thyroid diseases are also reported to be associated with DVT. In a large prospective cohort study [ 31 ], the incidence of venous thromboembolism was observed to increase in patients with thyroid cancer over the age of 60. However, other retrospective studies did not find any difference compared with the general population [ 32 ]. In the post-COVID-19 era, subacute thyroiditis has received considerable attention from researchers. New evidence suggests that COVID-19 may be associated with subacute thyroiditis [ 33 , 34 ]. Mondal et al. [ 35 ] found that out of 670 COVID-19 patients, 11 presented with post-COVID-19 subacute thyroiditis. Among them, painless subacute thyroiditis appeared earlier and exhibited symptoms of hyperthyroidism. Another case report also indicated the same result, that is, subacute thyroiditis occurred after COVID-19 infection, accompanied by thyroid function changes [ 36 ]. This led us to hypothesize that subacute thyroiditis may cause DVT through alterations in thyroid function.

This study confirmed a significant causal relationship between autoimmune hyperthyroidism and DVT ( p  = 0.02). The data were tested for heterogeneity and gene pleiotropy using MR-Egger, Cochran’s Q, and MR-PRESSO tests. There was no evidence that the results were influenced by pleiotropy or heterogeneity. In the leave-one-out analysis, four of the five selected SNPs showed significant effects of autoimmune hyperthyroidism on DVT, suggesting an impact of these SNPs on DVT outcome. Previous studies have focused on the relationship between hyperthyroidism and its secondary arrhythmias and arterial thromboembolism [ 37 , 38 ]. This study emphasized the risk of DVT in patients with hyperthyroidism, which has certain clinical implications. Prophylactic anticoagulant therapy was observed to help prevent DVT in patients with hyperthyroidism. Unfortunately, the results of this study did not reveal any evidence that suggests a relationship between other thyroid diseases and DVT occurrence. This may be due to the limited database, as this study only included the GWAS data from a subset of European populations. Large-scale multiracial studies are needed in the future.

There are some limitations to this study. First, it was limited to participants of European descent. Consequently, further investigation is required to confirm these findings in other ethnicities. Second, this study did not reveal the relationship between complications of hyperthyroidism and DVT. Additionally, this study selected IVs from the database using statistical methods rather than selecting them from the real population. This may result in weaker effects of the screened IVs and reduce the clinical significance of MR analysis. Moreover, the definitions of some diseases in this study were not clear in the original database, and some of the diseases were self-reported, which may reduce the accuracy of diagnosis. Further research is still needed to clarify the causal relationship between DVT and thyroid diseases based on prospective cohort and randomized controlled trials (RCTs).

This study analyzed large-scale genetic data and provided evidence of a causal relationship between autoimmune hyperthyroidism and the risk of DVT, Compared with the other thyroid diseases investigated. Prospective RCTs or MR studies with larger sample sizes are still needed to draw more precise conclusions.

Availability of data and materials

The IEU open gwas project, https://gwas.mrcieu.ac.uk/

Ortel TL, Neumann I, Ageno W, et al. American society of hematology 2020 guidelines for management of venous thromboembolism: treatment of deep vein thrombosis and pulmonary embolism. Blood Adv. 2020;4(19):4693–738.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mehrabi F, Farshbafnadi M, Rezaei N. Post-discharge thromboembolic events in COVID-19 patients: a review on the necessity for prophylaxis. Clin Appl Thromb Hemost. 2023;29:10760296221148476.

Article   PubMed   PubMed Central   Google Scholar  

Loffredo L, Vidili G, Sciacqua A, et al. Asymptomatic and symptomatic deep venous thrombosis in hospitalized acutely ill medical patients: risk factors and therapeutic implications. Thromb J. 2022;20(1):72.

RIETE Registry. Death within 30 days. RIETE Registry. 2022[2023.8.23]. https://rieteregistry.com/graphics-interactives/dead-30-days/ .

Minges KE, Bikdeli B, Wang Y, Attaran RR, Krumholz HM. National and regional trends in deep vein thrombosis hospitalization rates, discharge disposition, and outcomes for medicare beneficiaries. Am J Med. 2018;131(10):1200–8.

Di Nisio M, van Es N, Büller HR. Deep vein thrombosis and pulmonary embolism. Lancet. 2016;388(10063):3060–73.

Article   PubMed   Google Scholar  

Aquila I, Boca S, Caputo F, et al. An unusual case of sudden death: is there a relationship between thyroid disorders and fatal pulmonary thromboembolism? A case report and review of literature. Am J Forensic Med Pathol. 2017;38(3):229–32.

Katić J, Katić A, Katić K, Duplančić D, Lozo M. Concurrent deep vein thrombosis and pulmonary embolism associated with hyperthyroidism: a case report. Acta Clin Croat. 2021;60(2):314–6.

PubMed   PubMed Central   Google Scholar  

Hieber M, von Kageneck C, Weiller C, Lambeck J. Thyroid diseases are an underestimated risk factor for cerebral venous sinus thrombosis. Front Neurol. 2020;11:561656.

Pohl KR, Hobohm L, Krieg VJ, et al. Impact of thyroid dysfunction on short-term outcomes and long-term mortality in patients with pulmonary embolism. Thromb Res. 2022;211:70–8.

Article   CAS   PubMed   Google Scholar  

Sirota DK. Axillary vein thrombosis as the initial symptom in metastatic papillary carcinoma of the thyroid. Mt Sinai J Med. 1989;56(2):111–3.

CAS   PubMed   Google Scholar  

Raveh E, Cohen M, Shpitzer T, Feinmesser R. Carcinoma of the thyroid: a cause of hypercoagulability? Ear Nose Throat J. 1995;74(2):110–2.

Davey Smith G, Hemani G. Mendelian randomization: genetic anchors for causal inference in epidemiological studies. Hum Mol Genet. 2014;23(R1):R89–98.

Stuijver DJ, van Zaane B, Romualdi E, Brandjes DP, Gerdes VE, Squizzato A. The effect of hyperthyroidism on procoagulant, anticoagulant and fibrinolytic factors: a systematic review and meta-analysis. Thromb Haemost. 2012;108(6):1077–88.

PubMed   Google Scholar  

Son HM. Massive cerebral venous sinus thrombosis secondary to Graves’ disease. Yeungnam Univ J Med. 2019;36(3):273–80.

Elbers LP, Moran C, Gerdes VE, et al. The hypercoagulable state in hyperthyroidism is mediated via the thyroid hormone β receptor pathway. Eur J Endocrinol. 2016;174(6):755–62.

Davis PJ, Sudha T, Lin HY, et al. Thyroid hormone, hormone analogs, and angiogenesis. Compr Physiol. 2015;6(1):353–62.

Mousa SA, Lin HY, Tang HY, et al. Modulation of angiogenesis by thyroid hormone and hormone analogues: implications for cancer management. Angiogenesis. 2014;17(3):463–9.

Lou Z, Li X, Li C, et al. Microarray profile of circular RNAs identifies hsa_circ_000455 as a new circular RNA biomarker for deep vein thrombosis. Vascular. 2022;30(3):577–89.

Hemani G, Bowden J, Davey SG. Evaluating the potential role of pleiotropy in Mendelian randomization studies. Hum Mol Genet. 2018;27(R2):R195–208.

Zhang Z, Li L, Hu Z, et al. Causal effects between atrial fibrillation and heart failure: evidence from a bidirectional Mendelian randomization study. BMC Med Genomics. 2023;16(1):187.

Emdin CA, Khera AV, Kathiresan S. Mendelian randomization. JAMA. 2017;318(19):1925–6.

Skrivankova VW, Richmond RC, Woolf BAR, et al. Strengthening the reporting of observational studies in epidemiology using Mendelian randomization: the STROBE-MR statement. JAMA. 2021;326(16):1614–21.

Hemani G, Zheng J, Elsworth B, et al. The MR-Base platform supports systematic causal inference across the human phenome. Elife. 2018;7: e34408.

Verbanck M, Chen CY, Neale B, Do R. Detection of widespread horizontal pleiotropy in causal relationships inferred from Mendelian randomization between complex traits and diseases. Nat Genet. 2018;50(5):693–8.

Franchini M. Hemostatic changes in thyroid diseases: haemostasis and thrombosis. Hematology. 2006;11(3):203–8.

Franchini M, Lippi G, Targher G. Hyperthyroidism and venous thrombosis: a casual or causal association? A systematic literature review. Clin Appl Thromb Hemost. 2011;17(4):387–92.

Fandler-Höfler S, Pilz S, Ertler M, et al. Thyroid dysfunction in cerebral venous thrombosis: a retrospective cohort study. J Neurol. 2022;269(4):2016–21.

Krieg VJ, Hobohm L, Liebetrau C, et al. Risk factors for chronic thromboembolic pulmonary hypertension—importance of thyroid disease and function. Thromb Res. 2020;185:20–6.

Alsaidan AA, Alruwiali F. Association between hyperthyroidism and thromboembolism: a retrospective observational study. Ann Afr Med. 2023;22(2):183–8.

Walker AJ, Card TR, West J, Crooks C, Grainge MJ. Incidence of venous thromboembolism in patients with cancer—a cohort study using linked United Kingdom databases. Eur J Cancer. 2013;49(6):1404–13.

Ordookhani A, Motazedi A, Burman KD. Thrombosis in thyroid cancer. Int J Endocrinol Metab. 2017;16(1): e57897.

Ziaka M, Exadaktylos A. Insights into SARS-CoV-2-associated subacute thyroiditis: from infection to vaccine. Virol J. 2023;20(1):132.

Henke K, Odermatt J, Ziaka M, Rudovich N. Subacute thyroiditis complicating COVID-19 infection. Clin Med Insights Case Rep. 2023;16:11795476231181560.

Mondal S, DasGupta R, Lodh M, Ganguly A. Subacute thyroiditis following recovery from COVID-19 infection: novel clinical findings from an Eastern Indian cohort. Postgrad Med J. 2023;99(1172):558–65.

Nham E, Song E, Hyun H, et al. Concurrent subacute thyroiditis and graves’ disease after COVID-19: a case report. J Korean Med Sci. 2023;38(18): e134.

Mouna E, Molka BB, Sawssan BT, et al. Cardiothyreosis: epidemiological, clinical and therapeutic approach. Clin Med Insights Cardiol. 2023;17:11795468231152042.

Maung AC, Cheong MA, Chua YY, Gardner DS. When a storm showers the blood clots: a case of thyroid storm with systemic thromboembolism. Endocrinol Diabetes Metab Case Rep. 2021;2021:20–0118.

Download references

Not applicable.

Author information

Lifeng Zhang and Kaibei Li have contributed equally to this work and share the first authorship.

Authors and Affiliations

Department of Vascular Surgery, Hospital of Chengdu University of Traditional Chinese Medicine, No. 39, Shierqiao Road, Jinniu District, Chengdu, 610072, Sichuan, People’s Republic of China

Lifeng Zhang, Qifan Yang, Yao Lin, Caijuan Geng, Wei Huang & Wei Zeng

Disinfection Supply Center, Hospital of Chengdu University of Traditional Chinese Medicine, No. 39, Shierqiao Road, Jin Niu District, Chengdu, 610072, Sichuan, People’s Republic of China

You can also search for this author in PubMed   Google Scholar

Contributions

Conception and design: LFZ and WZ. Analysis and interpretation: LFZ, KBL and WZ. Data collection: LFZ, QFY, YL, CJG and WH. Writing the article: LFZ, KBL. Critical revision of the article: LFZ, GFY and WZ. Final approval of the article: LFZ, KBL, YL, CJG, WH, QFY and WZ. Statistical analysis: YL, QFY.

Corresponding author

Correspondence to Wei Zeng .

Ethics declarations

Ethics approval and consent to participate.

Ethical approval was obtained in all original studies. This study complies with the terms of use of the database.

Competing interests

Additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Zhang, L., Li, K., Yang, Q. et al. Associations between deep venous thrombosis and thyroid diseases: a two-sample bidirectional Mendelian randomization study. Eur J Med Res 29 , 327 (2024). https://doi.org/10.1186/s40001-024-01933-1

Download citation

Received : 12 September 2023

Accepted : 09 June 2024

Published : 14 June 2024

DOI : https://doi.org/10.1186/s40001-024-01933-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Deep venous thrombosis
• Thyroid diseases
• Mendelian randomization analysis

European Journal of Medical Research

ISSN: 2047-783X

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 15 June 2024

Outcomes of different pulmonary rehabilitation protocols in patients under mechanical ventilation with difficult weaning: a retrospective cohort study

• Shiauyee Chen 1 , 2   na1 ,
• Shu-Fen Liao 3 , 4   na1 ,
• Yun-Jou Lin 2 , 5 ,
• Chao-Ying Huang 2 , 6 ,
• Shu-Chuan Ho 2 , 7 &
• Jer-Hwa Chang 2 , 8  

Respiratory Research volume  25 , Article number:  243 ( 2024 ) Cite this article

128 Accesses

Metrics details

The endeavor of liberating patients from ventilator dependence within respiratory care centers (RCCs) poses considerable challenges. Multiple factors contribute to this process, yet establishing an effective regimen for pulmonary rehabilitation (PR) remains uncertain. This retrospective study aimed to evaluate existing rehabilitation protocols, ascertain associations between clinical factors and patient outcomes, and explore the influence of these protocols on the outcomes of the patients to shape suitable rehabilitation programs.

Conducted at a medical center in northern Taiwan, the retrospective study examined 320 newly admitted RCC patients between January 1, 2015, and December 31, 2017. Each patient received a tailored PR protocol, following which researchers evaluated weaning rates, RCC survival, and 3-month survival as outcome variables. Analyses scrutinized differences in baseline characteristics and prognoses among three PR protocols: protocol 1 (routine care), protocol 2 (routine care plus breathing training), and protocol 3 (routine care plus breathing and limb muscle training).

Among the patients, 28.75% followed protocol 1, 59.37% protocol 2, and 11.88% protocol 3. Variances in age, body-mass index, pneumonia diagnosis, do-not-resuscitate orders, Glasgow Coma Scale scores (≤ 14), and Acute Physiology and Chronic Health Evaluation II (APACHE) scores were notable across these protocols. Age, APACHE scores, and abnormal blood urea nitrogen levels (> 20 mg/dL) significantly correlated with outcomes—such as weaning, RCC survival, and 3-month survival. Elevated mean hemoglobin levels linked to increased weaning rates ( p  = 0.0065) and 3-month survival ( p  = 0.0102). Four adjusted models clarified the impact of rehabilitation protocols. Notably, the PR protocol 3 group exhibited significantly higher 3-month survival rates compared to protocol 1, with odds ratios (ORs) ranging from 3.87 to 3.97 across models. This association persisted when comparing with protocol 2, with ORs between 3.92 and 4.22.

Our study showed that distinct PR protocols significantly affected the outcomes of ventilator-dependent patients within RCCs. The study underlines the importance of tailored rehabilitation programs and identifies key clinical factors influencing patient outcomes. Recommendations advocate prospective studies with larger cohorts to comprehensively assess PR effects on RCC patients.

Respiratory care centers (RCCs), specialized units downstream of ICUs in Taiwan, provide effective care for patients who have repeatedly experienced ventilator weaning failure while in intensive care and have undergone prolonged mechanical ventilation [ 1 ]. In previous investigations, weaning success rates ranged from 38–70% [ 2 , 3 , 4 , 5 , 6 , 7 ]. Successful weaning has been attributed to several factors including age, nutritional status, comorbid conditions, muscle strength, lung mechanics, renal function, and hemoglobin (Hb) levels [ 1 , 2 , 8 , 9 , 10 ]. For adult patients on prolonged mechanical ventilation (PMV), factors associated with unsuccessful weaning include a longer duration of RCC stay, an elevated blood urea nitrogen (BUN) level, a lower modified Glasgow Coma Scale (GCS) score, and lower serum albumin and maximal inspiratory pressure (PI max ) levels [ 11 , 12 ]. Many PMV patients are discharged from the hospital but are again readmitted within one year [ 13 ]; Stoller et al’s study found that patients who were discharged from the weaning unit had a mortality rate that fell by 68% within the first 2 years [ 14 ]. There were better survival outcomes in PMV patients with the following characteristics: no comorbidities, a tracheostomy, successful weaning, and aged less than 75 years [ 15 , 16 ].

Although the evidence supports that early rehabilitation can decrease ventilation duration for patients under weaning, the type and frequency of intervention are quite diverse. Prolonged weaning may impact patient’s muscular and skeletal systems. This aggregates the deconditioning even after weaning from the ventilator. Thus, early rehabilitation is important and have been reportedly related to decreased morbidity and mortality [ 17 ], disease complications [ 18 ], duration of the ICU stay, duration of the hospital stay [ 19 , 20 ], the re-hospitalization rate [ 21 ], and improved in physical function [ 22 ]. Additionally, several studies show that early rehabilitation can improve weaning rate for patients with prolonged mechanical ventilation. Among the studies, treatment types are various and can be classified into conventional physical therapy, exercise-based physical therapy, neuromuscular electrical stimulation (NMES), progressive mobility, inspiratory muscle training and multi-component training programs. The training frequencies are varied from high (over 2 sessions/day), moderate (one session/day, 3–7 days/week) to low (one session/day, less than 3 days a week) [ 23 , 24 , 25 , 26 ].

Even with integrated care, successful weaning is difficult and related to multiple factors. The appropriate regimen of rehabilitation protocols has not been confirmed. The efficacy of pulmonary rehabilitation (PR) has been proven through random controlled trials (RCTs) and systematic meta-analyses, but implementation of PR in the real world still encounters barriers such as restricted resources or eligibility of patients. Thus, the aims of this retrospective study were to (1) review the status of rehabilitation protocols and provide information for suitable programs, (2) find associations of clinical factors with patients’ outcomes, and (3) determine the effects of rehabilitation protocols on outcomes in RCC units.

Study design and data sources

This was a retrospective chart review study. The study was approved by the TMU-Joint Institutional Review Board with approval no. N201612048. Patients (over 20 years of age) admitted to an RCC (a semi-ICU) from January 1, 2015 to December 31, 2017 were recruited. The RCC unit was located in a medical center in northern Taiwan (a single-center study). We gained access to the patient list and charts from our secretary department. We used traditional PR recording books and an electronic chart system for data collection [ 27 , 28 , 29 ].

Patients were transferred from our medical ICU, surgical ICU, general ward, or emergent department. Patients with intermittent mandatory ventilation (IMV) could be tracheotomized or have an endotracheal tube. Patients received a multidisciplinary approach after admission. Team members included chest and critical care physicians, nurses, respiratory therapists, physical therapists, dietitians, and social workers.

Weaning protocol

The weaning protocol for patients was discussed by team members at a bedside meeting in the morning. Patients had to meet certain criteria before attempting to wean: no hemodynamic instability; no vasopressor therapy; systolic blood pressure (SBP) of > 100 mmHg; heart rate range of 50 to 130 beats per minute; no fever; fraction of inspired oxygen of < 40%; and positive end expiratory pressure (PEEP) of < 8 cmH 2 O. Ventilatory support was gradually reduced to a level of Synchronized intermittent mandatory ventilation (SIMV), and then the level of pressure-supported ventilation (PSV) was reduced. The duration of these trials was gradually increased from 2 to 24 h. Patients were extubated when they passed the 30 min to 2-h spontaneous breathing trial and arterial blood gas (ABG) checkup.

The protocol was comprised of several steps [ 11 , 30 ]. In patients with non-invasive ventilation (NIV), the time of support was gradually reduced with the oxygen supply via an oxygen mask or nasal cannula. The duration of these trials was gradually increased (beginning at 2 h and finishing at 24 h). Success in weaning was defined as 5 days of complete liberation from the ventilator.

Intervention of rehabilitation

When patients were transferred to the RCC, doctors would refer the patients for physical therapy (PT). Three well-trained physical therapists with more than 3 years working in RCC conducted a rehabilitation intervention according to the algorithm of the PR protocol (Fig.  1 ). The PT programs were held when the patients were under one of the following situations: (1) fraction of inspired oxygen (FiO 2 ) of ≥ 50% or PEEP of ≥ 8 cmH 2 O, or blood oxygen saturation level (SpO 2 ) of ≤ 90%; (2) SBP of ≤ 90 or ≥ 200 mmHg; (3) respiratory rate (RR) of ≥ 35/min; (4) heart rate (HR) of ≤ 50 or ≥ 130/min; (5) a new episode of arrhythmia, e.g., ventricular premature contraction (VPC) bigeminy, bradycardia; (6) under support of a vasopressor, e.g., norepinephrine > 4 µg/min or dopamine > 5 mg/kg/min; (7) new episodes of acute myocardial infarction or chest tightness; (8) under hemodialysis (H/D); and (9) gastrointestinal (GI) bleeding [ 31 , 32 ].

Protocols for Pulmonary Rehabilitation in the Respiratory Care Unit (RCC).

PT interventions provided one session a day and five sessions a week. A session lasted 15 ~ 50 min depending on the applied programs and the condition of the patient. Before the exercise intervention, PT evaluated the patients and confirmed the following items with team members: weaning plan, hemodynamics, feeding, and contraindications. The level of conscious and the ability to follow orders were also considered. PT discussed with team members and assigned patients to a specific protocol. Combinations of PT interventions depended on the patients’ conditions and PT assessments.” The pulmonary rehabilitation protocols included (1) PR protocol 1: routine care (active assisted range of motion exercise, or passive range of motion exercise, positioning, and tilt up); (2) PR protocol 2: routine care plus breathing training (deep breathing exercise and pursed-lip breathing exercise or abdominal binder or inspiratory muscle training; and (3) PR protocol 3: routine care plus breathing training and limb muscle training (ergometer bike training for lower extremities, or Thera-band resistive training for the upper extremities, or strength exercise for the lower extremities) [ 33 , 34 ].

We began inspiratory muscle training when the patient was under the support of the following modes: PC/SIMV + PS or under PSV. When a patient was under the support of continuous positive airway pressure (CPAP) or oxygen support with a nasal cannula or collar, PT would hold inspiratory muscle training. The feeding schedule was coordinated with the training program. Training started 30 min to 1 h after feeding.

An abdominal binder was placed between the area of the lower rib cage and anterior superior iliac crest. The binder’s upper edge was below the costal margin so that it minimally interfered with ribcage movement. In consideration of digestion and skin allergies, the binder was released for 1 h after feeding and during the night [ 35 , 36 ].

Measurements

Primary outcomes were the weaning status, RCC survival, and 3-month survival after RCC discharge. Recorded data were as follows: demographics (age, sex, and body-mass index (BMI)), type of ventilator (IMV or NIV), rehabilitation intervention, laboratory data at RCC admission (BUN and Hb), a do-not-resuscitate (DNR) order (signed or not), a diagnosis of pneumonia, status of blood pressure at admission, Acute Physiology and Chronic Health Evaluation II (APACHE) score at RCC discharge, and GCS score at RCC entry. Patients with SBP of ≤ 90 mmHg or diastolic blood pressure (DBP) of ≤ 60 mmHg were defined as having hypotension. BUN of > 20 mg/dL and a GCS score of ≤ 14 were further applied.

Statistical analysis

We evaluated the effects of PR protocols on prognosis including the weaning rate, RCC survival, and 3-month survival. Differences in baseline demographic characteristics and prognoses among the three PR protocols were assessed using a Chi-squared test for categorical variables and an analysis of variance (ANOVA) for continuous variables. A logistic regression model was applied to estimate odds ratios (ORs) and 95% confidence intervals (CIs) for associations between PR protocols and prognoses.

Confounding by indication bias may have been present in the study because the rehabilitation intervention may have been implemented in patients depending on their baseline characteristics, comorbidities, or clinical factors during RCC hospitalization, including age, sex, pneumonia, DNR, ventilator type, and GCS score. A propensity score, first proposed by Rosenbaum and Rubin [ 37 ], was estimated to present the probabilities of patients adopting different PR protocols. The propensity score weighting (PSW) procedure with an inverse probability of a treatment weighting approach was applied to create pseudo-populations for the three different PR protocols to account for the bias [ 38 ]. Imbalances in characteristics of indications among PR groups were well controlled in the pseudo-populations, resulting in virtual randomization of the rehabilitation intervention [ 39 ]. The pseudo-populations were used for subsequent analyses. Associations of rehabilitation with weaning, survival, and 3-month survival were examined through crude and four multiple-adjusted models. For model 1, we adjusted for sex and age only. Characteristics significantly associated with each prognosis variable in the univariate analyses were considered potential confounders and were adjusted for in multiple regression models 2 and 3. Because information on APACHE score at admission was mostly unknown in our study, we developed two different models with and without including APACHE scores. Finally, all putative confounders except for APACHE scores were adjusted for in model 4.

Statistical analyses were two-sided, and the level of significance was 0.05. All statistical analyses were performed using SAS software (vers. 9.4; SAS Institute, Cary, NC, USA).

In the study, we included 320 patients admitted to an RCC to evaluate the association between their rehabilitation protocol and their prognosis. The weaning rate and RCC survival (there was one patient with an unknown RCC survival status) were 57.50% and 89.06%, respectively. The 3-month survival status was ascertained for 236 patients (73.75% of all patients), showing a 3-month survival rate of 59.75%. Demographic characteristics of patients are presented in Table  1 . All patients adopted routine care. There were 92 (28.75%), 190 (59.37%), and 38 (11.88%) patients who adopted PR protocol 1, 2, and 3, respectively. Distributions of age, BMI, a pneumonia diagnosis, having signed a DNR order, with a GCS score of ≤ 14, and an APACHE score were significantly differed among the three rehabilitation protocols. The mean age of patients receiving PR protocol 3 (73.79; SD = 13.52 years) was the youngest compared to PR protocol 1 (80.44; SD = 13.30 years) and PR protocol 2 (80.67; SD = 11.56 years). The mean BMI and APACHE score were highest for PR protocol 3 compared to PR protocol 1 and 2. There were smaller proportions of patients with pneumonia, who had signed a DNR order, and with a GCS score of ≤ 14 in the PR protocol 3 group. The proportions using IMV in the three groups were 47.83%, 57.89%, and 42.11%, respectively, with comparable distributions in the mode of ventilation. No significant differences in hypotension, BUN of > 20 mg/dL, and Hb were detected among the three PR protocols. The average duration of PT for all patients was 10.59 days (PR protocol 1: 8.00 days, PR.

protocol 2: 12.04 days, and PR protocol 3: 9.62 days).

Propensity score-weighted cohort

In order to reduce confounding by indication bias due to an imbalance of rehabilitation protocol assignments, we performed a propensity score-weighting approach in this study. Age, sex, pneumonia, DNR, ventilator type, and GCS were used to calculate propensity scores. After the weighting procedure, the weighted cohort comprised 315.28 patients in total, with 89.82, 189.84, and 35.62 patients in PR protocols 1, 2, and 3, respectively (Table  2 ). All variables included in the propensity score were balanced in the three rehabilitation protocols among the weighted cohort.

Rehabilitation, demographic characteristics, and prognosis

Distributions of rehabilitation and demographic characteristics of weaning, survival, and 3-month survival among the weighted cohort are demonstrated in Table  3 . Rehabilitation showed a borderline significant effect on 3-month survival ( p  = 0.0554), but not on the weaning rate or RCC survival. We found that 58.65%, 56.83%, and 84.61% of patients adopting PR protocol 1, 2, and 3 were alive at 3 months, respectively. Age, APACHE score, and an abnormal BUN level (> 20 mg/dL) were significantly associated with the prognosis, including weaning, RCC survival, and 3-month survival. Signing a DNR order and ventilator type were also critical factors related to RCC survival and 3-month survival. Patients diagnosed with pneumonia or detected with hypotension presented a poorer prognosis than patients without the corresponding disease (pneumonia: 87.55% vs. 97.16% for RCC survival; hypotension: 52.32% vs. 70.69% for 3-month survival). A higher mean Hb was associated with a higher weaning rate ( p  = 0.0065) and 3-month survival ( p  = 0.0102).

Effects of rehabilitation protocols on the prognosis

To investigate associations of rehabilitation protocols with weaning, survival, and 3-month survival, a univariate analysis and several adjusted models were performed in the study (Table  4 ). In the univariate analysis, there were no significant associations between rehabilitation protocols and weaning or RCC survival. Patients adopting PR protocol 3 (OR: 3.87; 95% CI: 1.08–13.90; p  = 0.0378) showed a better 3-month survival compared to those adopting PR protocol 1, but the beneficial effect was not presented for the PR protocol 2 group.

We further applied four adjusted models to clarify the effect of rehabilitation protocols, including model 1 (adjusted for sex and age), model 2 (adjusted for all variables except the APACHE score that reached statistical significance in the univariate analysis), model 3 (adjusted for all variables that reached statistical significance in the univariate analysis), and model 4 (adjusted for sex, age, DNR, ventilator type, hypotension, BUN, pneumonia, GCS, and Hb). Results were similar to those shown in the univariate analysis (Table  4 ). PR protocol 2 was not significantly associated with weaning, survival, or 3-month survival, compared to PR protocol 1. According to models 1 and 2, 3-month survival in the PR protocol 3 group was significantly higher than that in the PR protocol 1 group, with ORs of 3.89 (95% CI: 1.06–14.23, p  = 0.04) and 3.97 (95% CI: 1.02–15.43, p  = 0.0469), respectively. The significant association remained when taking the PR protocol 2 group as the reference, with ORs of 3-month survival in the PR protocol 3 group of 4.17 (95% CI: 1.22–14.31, p  = 0.0230), 4.22 (95% CI: 1.21–14.74, p  = 0.0240), and 3.92 (95% CI: 1.07–14.39, p  = 0.0394) under the univariate analysis, model 1, and model 2, respectively. The beneficial effect of PR protocol 3 (compared to PR protocol 1) became borderline significant or non-significant under models 4 and 3, possibly because of insufficient power in the analyses.

This retrospective chart review study showed that different pulmonary rehabilitation protocols were associated with 3-month survival after discharge from an RCC, but not the weaning status or survival in RCC hospitalization. Patients who received routine care plus respiratory muscle and limb muscle training had an OR of 3.87 in the crude model compared to routine care for the outcome of 3-month survival (with a survival rate of 84.61%).

In a retrospective analysis over 10 years [ 40 ], 180 tracheostomized patients received PR programs during weaning at a semi-ICU. The individual rehabilitation programs (IRPs) included intensive physiotherapy (peripheral and respiratory muscle reconditioning, airway secretion, and breathing exercises). The PR programs were executed twice a day for 1 h each time. The programs were adjusted according to a patient’s clinical stability. Results showed that the level of comorbidities (Cumulative Illness Rating Scale) was associated with weaning outcomes. The weaning rate for patients was 66.1%. In our study, our patients included those with an endotracheal tube ( n  = 76), those tracheostomized ( n  = 94), and those with NIV ( n  = 150), and the weaning rate for all subjects was 57.50%. The PR programs in our study were also tailored by our team members, but only once daily. Our programs provided three main components: routine care, breathing training, and limb muscle training. Not all of the patients received all components. Our study also found that the APACHE score was significantly associated with weaning outcomes. Since the APACHE score is a general measure of disease severity, our study had similar findings to a study by Costi et al. [ 34 ] Under implementation of PR, the weaning outcome was still influenced by the level of severity.

Keng et al. [ 41 ] found that the post-rehabilitation functional status was independently associated with the weaning success, 3-month survival after RCC discharge, and hospital survival. The de Morton Morbidity Index (DEMMI) was selected as a tool to measure the responsiveness to PR rather than the acceptance of PR itself. Patients in this study were under prolonged mechanical ventilation and received standard sessions of the training program. Due to concern with patient tolerance and hemodynamic stability, we assigned patients to appropriate regimens in our study after discussion with team members. Maybe that resulted in a lower intensity or duration which would have impacted weaning and RCC survival. Patient who were rehabilitation responders (post-rehab DEMMI of ≥ 20) had significantly higher weaning success and survival at hospital discharge and at 3 months after RCC discharge regardless of the pre-rehab DEMMI score. That supports the importance of PR protocols in the unit focused on weaning. Our study revealed that patients who adopted PR protocol 3 showed a better 3-month survival compared to those who adopted PR protocol 1 (OR: 3.87). Patients in PR protocol 3 received limb muscle training, which should have been beneficial for functional recovery. For those poor-responders to rehabilitation, clinical outcomes may have been limited by other factors. Results of our study found that age, APACHE score, BUN level, signing a DNR order, ventilator type, a pneumonia diagnosis, hypotension, and Hb level were associated with weaning, RCC survival, or 3-month survival.

Controversies in outcomes may depend on the subject eligibility criteria, content of the programs, and organization’s facilities [ 22 , 23 , 25 ]. Verceles’s study applied multimodal rehabilitation programs (MRP) for patients with PMV and ICU acquired weakness. Results showed that patients with usual care (UC) plus multimodal training programs (functional training: bed dependent to chair dependent or ambulatory programs) had greater weaning success, and more were discharged to home [ 23 ]. Our study had similar finding with this study. Patients under PR protocol 3 with limb muscle training had a higher 3-month survival rate (84.61%). The ages of subjects in Verceles’s study were 63.1 vs. 57.1 years (MRP + UC vs. UC). The pre-admission Barthel index scores were 90 vs. 96 (indicating a higher functional status). In case of a higher functional status before admission, subjects could tolerate intensive and functional mobility, strength, and endurance training [ 23 ]. The population in our subjects were older (79.79 ± 12.48 years). Since 85% of our subjects had a GCS score of ≤ 14, we could only apply routine care plus breathing training (abdominal binder or inspiratory muscle training) instead of limb muscle training or functional approaches.

We also discovered that younger patients were more quickly weaned from the ventilator, and maintained better RCC survival and 3-month survival. Past studies found that age had an independent effect on the result of patients with mechanical ventilation, such as ventilator dependence [ 42 ], complications, the duration of mechanical ventilation, and hospital stay [ 43 ].

Yang et al. found that APACHE II scores and serum albumin concentrations were the best weaning predictors in patients with pneumonia [ 1 ]. In our 78.75% of patients with pneumonia, we found that APACHE scores and serum BUN levels at RCC admission were significantly related to successful weaning, RCC survival, and 3-month survival. The mean APACHE score of reports of successfully weaned pneumonia patients was 16.9, while our mean APACHE score of 16.77 was consistent with those of Yang et al. Malnutrition may be an important factor in an RCC patient’s ability to be weaned from mechanical ventilation. Higher Hb levels above 10.2 g/dL seemed to increase the successful weaning rate [ 1 ]. In our study, we also found that higher hemoglobin levels above 10.13 g/dL seemed to increase the successful weaning rate and above 10.17 g/dL to increase 3-month survival.

Strengths and limitations

The strengths of our study included (1) offering different protocols for subjects, (2) defining contraindications and criteria to cease training, and (3) using propensity score weighting for the analysis to reduce confounding by indication bias due to imbalanced distributions in characteristics among the three different PR protocols. Results provide greater evidence because the study design was approximately like that of an RCT [ 38 ]. Limitations of our study included (1) a lack of pulmonary function measurement and a measure of the functional status before entry and discharge from the RCC such as Functional Independence Measure (FIM) or DEMMI. Pulmonary function has been considered a crucial variable for survival rates but it cannot be well controlled in the study. (2) This was a single study of a retrospective nature, which only reflected the experience in a single regional weaning unit, and so cannot be generalized to facilities in other regions. (3) Unavoidable data loss occurred for some parameters during clinical practice such as APACHE scores (missing for 44 patients) and 3-month survival data. This may have reduced the statistical power and resulted in an inability to replicate the significant association between PR protocols and 3-month survival in adjusted model 2 when APACHE scores were additionally included (model 3).

As to loss of data for 3-month survival, in the period of cohort studies, we could not easily obtain all of the outpatient data after discharge if patients did not return to our hospital. To evaluate the impact on our findings of data loss after discharge, we conducted a comparison of baseline characteristics and PR protocols between 3-month responders and non-responders among RCC survivors (supplementary Table 1 ). Similar distributions of the baseline characteristics in the two groups revealed the data loss was at random and the effect of PR protocols on 3-month survival obtained from the study was an under-estimate to true effect. In addition, we compared the 3-month survival according to the respiratory care ward (RCW) admission status after discharge from hospital among 3-month responders (supplementary Table 2 ) and found non-significant distribution in between.

From the data in our stduy, PR protocol in individual patient could be beneficial in the outcomes when we take associated factors into consideration. Early implementation and appropriate protocols are important for the patient with difficult weaning status. Ddifferent rehabilitation protocols, especially the protocol 3 including all three components of pulmonary rehabilitation programs as basic elements, breathing exercise and limb muscle training mostly benefit for patients under ventilator dependence, even for the long-termed survival rate. This might support an integrating rehabilitation program in future clinical field. However, being a retrospective study in a single weaning hospital unit as its inevitable limitations of our studies, it seems reasonable to conduct a prospective program to validate the significant effect of incorporated rehabilitation programs. Maybe in the future, these evidences would help clinical professionals for conducting rehabilitation programs for these patient population.

Data availability

No datasets were generated or analysed during the current study.

Abbreviations

Respiratory care centers

• Pulmonary rehabilitation

Acute Physiology and Chronic Health Evaluation II

Prolonged mechanical ventilation

Blood urea nitrogen

Glasgow Coma Scale

Multimodal rehabilitation program

Intermittent mandatory ventilation

Positive end expiratory pressure

Synchronized intermittent mandatory ventilation

Pressure supported ventilation

Arterial blood gas

Not-invasive ventilation

Physical therapy

Fraction of inspired oxygen

Continuous positive airway pressure

Do-not-resuscitate (DNR)

Propensity score weighting (PSW)

Yang PH, Hung JY, Yang CJ, Tsai JR, Wang TH, Lee JC, Huang MS. Successful weaning predictors in a respiratory care center in Taiwan. Kaohsiung J Med Sci. 2008;24:85–91.

Article   CAS   PubMed   Google Scholar  

Pilcher DV, Bailey MJ, Treacher DF, Hamid S, Williams AJ, Davidson AC. Outcomes, cost and long term survival of patients referred to a regional weaning centre. Thorax. 2005;60:187–92.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Carson SS, Bach PB, Brzozowski L, Leff A. Outcomes after long-term acute care. An analysis of 133 mechanically ventilated patients. Am J Respir Crit Care Med. 1999;159:1568–73.

Gracey DR, Hardy DC, Koenig GE. The chronic ventilator-dependent unit: a lower-cost alternative to intensive care. Mayo Clin Proc. 2000;75:445–9.

Schonhofer B, Guo JJ, Suchi S, Kohler D, Lefering R. The use of APACHE II prognostic system in difficult-to-wean patients after long-term mechanical ventilation. Eur J Anaesthesiol. 2004;21:558–65.

Pham T, Heunks L, Bellani G, Madotto F, Aragao I, Beduneau G, Goligher EC, Grasselli G, Laake JH, Mancebo J, et al. Weaning from mechanical ventilation in intensive care units across 50 countries (WEAN SAFE): a multicentre, prospective, observational cohort study. Lancet Respir Med. 2023;11:465–76.

Article   PubMed   Google Scholar  

Windisch W, Dellweg D, Geiseler J, Westhoff M, Pfeifer M, Suchi S, Schonhofer B. Prolonged weaning from mechanical ventilation. Dtsch Arztebl Int. 2020;117:197–204.

PubMed   PubMed Central   Google Scholar  

Sapijaszko MJ, Brant R, Sandham D, Berthiaume Y. Nonrespiratory predictor of mechanical ventilation dependency in intensive care unit patients. Crit Care Med. 1996;24:601–7.

Meade M, Guyatt G, Cook D, Griffith L, Sinuff T, Kergl C, Mancebo J, Esteban A, Epstein S. Predicting success in weaning from mechanical ventilation. Chest. 2001;120:S400–24.

Article   Google Scholar  

Na SJ, Ko RE, Nam J, Ko MG, Jeon K. Factors associated with prolonged weaning from mechanical ventilation in medical patients. Ther Adv Respir Dis. 2022;16:17534666221117005.

Article   PubMed   PubMed Central   Google Scholar  

Wu YK, Kao KC, Hsu KH, Hsieh MJ, Tsai YH. Predictors of successful weaning from prolonged mechanical ventilation in Taiwan. Respir Med. 2009;103:1189–95.

Trudzinski FC, Neetz B, Bornitz F, Muller M, Weis A, Kronsteiner D, Herth FJF, Sturm N, Gassmann V, Frerk T, et al. Risk factors for prolonged mechanical ventilation and weaning failure: a systematic review. Respiration. 2022;101:959–69.

MacIntyre NR, Epstein SK, Carson S, Scheinhorn D, Christopher K, Muldoon S. National Association for Medical Direction of Respiratory C: Management of patients requiring prolonged mechanical ventilation: report of a NAMDRC consensus conference. Chest 2005, 128:3937–3954.

Stoller JK, Xu M, Mascha E, Rice R. Long-term outcomes for patients discharged from a long-term hospital-based weaning unit. Chest. 2003;124:1892–9.

Huang C. Five years follow up of patient receiving prolonged mechanical ventilation: data for a single center in Taiwan. Front Med (Lausanne). 2022;9:1038915.

Warnke C, Heine A, Muller-Heinrich A, Knaak C, Friesecke S, Obst A, Bollmann T, Desole S, Boesche M, Stubbe B, Ewert R. Predictors of survival after prolonged weaning from mechanical ventilation. J Crit Care. 2020;60:212–7.

Combes A, Costa MA, Trouillet JL, Baudot J, Mokhtari M, Gibert C, Chastre J. Morbidity, mortality, and quality-of-life outcomes of patients requiring > or = 14 days of mechanical ventilation. Crit Care Med. 2003;31:1373–81.

Fuke R, Hifumi T, Kondo Y, Hatakeyama J, Takei T, Yamakawa K, Inoue S, Nishida O. Early rehabilitation to prevent postintensive care syndrome in patients with critical illness: a systematic review and meta-analysis. BMJ Open. 2018;8:e019998.

Klein K, Mulkey M, Bena JF, Albert NM. Clinical and psychological effects of early mobilization in patients treated in a neurologic ICU: a comparative study. Crit Care Med. 2015;43:865–73.

Needham DM, Korupolu R, Zanni JM, Pradhan P, Colantuoni E, Palmer JB, Brower RG, Fan E. Early physical medicine and rehabilitation for patients with acute respiratory failure: a quality improvement project. Arch Phys Med Rehabil. 2010;91:536–42.

Morris PE, Griffin L, Berry M, Thompson C, Hite RD, Winkelman C, Hopkins RO, Ross A, Dixon L, Leach S, Haponik E. Receiving early mobility during an intensive care unit admission is a predictor of improved outcomes in acute respiratory failure. Am J Med Sci. 2011;341:373–7.

Morris PE, Berry MJ, Files DC, Thompson JC, Hauser J, Flores L, Dhar S, Chmelo E, Lovato J, Case LD, et al. Standardized Rehabilitation and Hospital length of stay among patients with Acute Respiratory failure: a Randomized Clinical Trial. JAMA. 2016;315:2694–702.

Verceles AC, Wells CL, Sorkin JD, Terrin ML, Beans J, Jenkins T, Goldberg AP. A multimodal rehabilitation program for patients with ICU acquired weakness improves ventilator weaning and discharge home. J Crit Care. 2018;47:204–10.

Schreiber AF, Ceriana P, Ambrosino N, Malovini A, Nava S. Physiotherapy and Weaning from prolonged mechanical ventilation. Respir Care. 2019;64:17–25.

Wu RY, Yeh HJ, Chang KJ, Tsai MW. Effects of different types and frequencies of early rehabilitation on ventilator weaning among patients in intensive care units: a systematic review and meta-analysis. PLoS ONE. 2023;18:e0284923.

Bissett B, Gosselink R, van Haren FMP. Respiratory muscle Rehabilitation in patients with prolonged mechanical ventilation: a targeted Approach. Crit Care. 2020;24:103.

Vassar M, Holzmann M. The retrospective chart review: important methodological considerations. J Educ Eval Health Prof. 2013;10:12.

Siems A, Banks R, Holubkov R, Meert KL, Bauerfeld C, Beyda D, Berg RA, Bulut Y, Burd RS, Carcillo J, et al. Structured Chart Review: Assessment of a structured Chart Review Methodology. Hosp Pediatr. 2020;10:61–9.

Dziadkowiec O, Durbin J, Muralidharan VJ, Novak M, Cornett B. Improving the quality and design of Retrospective Clinical Outcome studies that Utilize Electronic Health Records. HCA Healthc J Med. 2020;1:131–8.

McConville JF, Kress JP. Weaning patients from the ventilator. N Engl J Med. 2012;367:2233–9.

Sommers J, Engelbert RH, Dettling-Ihnenfeldt D, Gosselink R, Spronk PE, Nollet F, van der Schaaf M. Physiotherapy in the intensive care unit: an evidence-based, expert driven, practical statement and rehabilitation recommendations. Clin Rehabil. 2015;29:1051–63.

Nordon-Craft A, Moss M, Quan D, Schenkman M. Intensive care unit-acquired weakness: implications for physical therapist management. Phys Ther. 2012;92:1494–506.

Clini EM, Crisafulli E, Antoni FD, Beneventi C, Trianni L, Costi S, Fabbri LM, Nava S. Functional recovery following physical training in tracheotomized and chronically ventilated patients. Respir Care. 2011;56:306–13.

Chen YH, Lin HL, Hsiao HF, Chou LT, Kao KC, Huang CC, Tsai YH. Effects of exercise training on pulmonary mechanics and functional status in patients with prolonged mechanical ventilation. Respir Care. 2012;57:727–34.

Abdallah SJ, Chan DS, Glicksman R, Mendonca CT, Luo Y, Bourbeau J, Smith BM, Jensen D. Abdominal binding improves neuromuscular efficiency of the human diaphragm during Exercise. Front Physiol. 2017;8:345.

Koo P, Gartman EJ, Sethi JM, McCool FD. Physiology in Medicine: physiological basis of diaphragmatic dysfunction with abdominal hernias-implications for therapy. J Appl Physiol (1985). 2015;118:142–7.

Brookhart MA, Schneeweiss S, Rothman KJ, Glynn RJ, Avorn J, Sturmer T. Variable selection for propensity score models. Am J Epidemiol. 2006;163:1149–56.

Austin PC. An introduction to Propensity score methods for reducing the effects of confounding in Observational studies. Multivar Behav Res. 2011;46:399–424.

Benedetto U, Head SJ, Angelini GD, Blackstone EH. Statistical primer: propensity score matching and its alternatives. Eur J Cardiothorac Surg. 2018;53:1112–7.

Costi S, Brogneri A, Bagni C, Pennacchi G, Beneventi C, Tabbi L, Dell’Orso D, Fantini R, Tonelli R, Beghi GM, Clini E. Rehabilitation of difficult-to-Wean, Tracheostomized patients admitted to Specialized Unit: retrospective analyses over 10-Years. Int J Environ Res Public Health 2022, 19.

Keng LT, Liang SK, Tseng CP, Wen YF, Tsou PH, Chang CH, Chang LY, Yu KL, Lee MR, Ko JC. Functional Status after Pulmonary Rehabilitation as a predictor of Weaning Success and Survival in patients requiring prolonged mechanical ventilation. Front Med (Lausanne). 2021;8:675103.

Ely EW, Evans GW, Haponik EF. Mechanical ventilation in a cohort of elderly patients admitted to an intensive care unit. Ann Intern Med. 1999;131:96–104.

Esteban A, Anzueto A, Frutos-Vivar F, Alia I, Ely EW, Brochard L, Stewart TE, Apezteguia C, Tobin MJ, Nightingale P, et al. Outcome of older patients receiving mechanical ventilation. Intensive Care Med. 2004;30:639–46.

Download references

Acknowledgements

The authors would like to thank all physical therapists who provided pulmonary rehabilitation programs for the patients and appreciate Dr.Kian Fan Chung for enhancing the quality of the work.

This study was funded by the Ministry of Science and Technology of Taiwan (MOST 111-2314-B-038-090-MY3, NSTC 112-2314-B-038-016; NSTC 112-2314-B-038-064). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author information

Shiauyee Chen and Shu-Fen Liao contributed equally to this work.

Authors and Affiliations

Department of Physical Medicine and Rehabilitation, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan

Shiauyee Chen

School of Respiratory Therapy, College of Medicine, Taipei Medical University, 250 Wuxing Street, Taipei, 11031, Taiwan

Shiauyee Chen, Yun-Jou Lin, Chao-Ying Huang, Shu-Chuan Ho & Jer-Hwa Chang

Department of Medical Research, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan

Shu-Fen Liao

School of Public Health, College of Public Health, Taipei Medical University, Taipei, Taiwan

Institute of Epidemiology and Preventive Medicine, College of Public Health, National Taiwan University, Taipei, Taiwan

Yun-Jou Lin

Graduate Institute of Physiology, College of Medicine, National Taiwan University, Taipei, Taiwan

Chao-Ying Huang

Division of Pulmonary Medicine, Department of Internal Medicine, Shuang Ho Hospital, Taipei Medical University, New Taipei City, Taiwan

Shu-Chuan Ho

Division of Pulmonary Medicine, Department of Internal Medicine, Wan Fang Hospital, Taipei Medical University, Taipei, Taiwan

Jer-Hwa Chang

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed substantially to the study design. SC, YJL and CYH collected data. SFL and SCH analyzed the data. SC and SCH wrote the first draft of the manuscript. SC, SFL, SCH and JHC critically reviewed and approved the final manuscript.

Corresponding authors

Correspondence to Shu-Chuan Ho or Jer-Hwa Chang .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Chen, S., Liao, SF., Lin, YJ. et al. Outcomes of different pulmonary rehabilitation protocols in patients under mechanical ventilation with difficult weaning: a retrospective cohort study. Respir Res 25 , 243 (2024). https://doi.org/10.1186/s12931-024-02866-3

Download citation

Received : 30 December 2023

Accepted : 04 June 2024

Published : 15 June 2024

DOI : https://doi.org/10.1186/s12931-024-02866-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mechanical ventilation with difficult weaning
• Ventilator weaning
• Respiratory care center

Respiratory Research

ISSN: 1465-993X

• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 20.6.2024 in Vol 10 (2024)

Digital Smoking Cessation Intervention for Cancer Survivors: Analysis of Predictors and Moderators of Engagement and Outcome Alongside a Randomized Controlled Trial

Authors of this article:

Original Paper

• Rosa Andree 1 , MSc   ; 
• Ajla Mujcic 2 , PhD   ; 
• Wouter den Hollander 1 , PhD   ; 
• Margriet van Laar 1 , PhD   ; 
• Brigitte Boon 3, 4, 5 , PhD   ; 
• Rutger Engels 6 , PhD   ; 
• Matthijs Blankers 1, 7 , PhD  

1 Trimbos Institute, Netherlands Institute of Mental Health and Addiction, Utrecht, Netherlands

2 PsyQ, Parnassia Groep, The Hague, Netherlands

3 Siza, Center for Long-term Care for People with Disabilities, Arnhem, Netherlands

4 Academy Het Dorp, Research & Advisory on Technology in Long-term Care, Arnhem, Netherlands

5 Tranzo, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, Netherlands

6 Erasmus School of Social and Behavioural Sciences, Erasmus University Rotterdam, Rotterdam, Netherlands

7 Department of Research, Arkin Mental Health Care, Amsterdam, Netherlands

Corresponding Author:

Rosa Andree, MSc

Trimbos Institute

Netherlands Institute of Mental Health and Addiction

Da Costakade 45

Utrecht, 3521 VS

Netherlands

Phone: 31 30 29 59 267

Email: [email protected]

Background: Recent studies have shown positive, though small, clinical effects of digital smoking cessation (SC) interventions for cancer survivors. However, research on associations among participant characteristics, intervention engagement, and outcomes is limited.

Objective: This study aimed to explore the predictors and moderators of engagement and outcome of MyCourse-Quit Smoking (in Dutch: “MijnKoers-Stoppen met Roken”), a digital minimally guided intervention for cancer survivors.

Methods: A secondary analysis of data from the randomized controlled trial was performed. The number of cigarettes smoked in the past 7 days at 6-month follow-up was the primary outcome measure. We analyzed interactions among participant characteristics (11 variables), intervention engagement (3 variables), and outcome using robust linear (mixed) modeling.

Results: In total, 165 participants were included in this study. Female participants accessed the intervention less often than male participants (B=–11.12; P =.004). A higher Alcohol Use Disorders Identification Test score at baseline was associated with a significantly higher number of logins (B=1.10; P <.001) and diary registrations (B=1.29; P <.001). A higher Fagerström Test for Nicotine Dependence score at baseline in the intervention group was associated with a significantly larger reduction in tobacco use after 6 months (B=–9.86; P =.002). No other associations and no moderating effects were found.

Conclusions: Overall, a limited number of associations was found between participant characteristics, engagement, and outcome, except for gender, problematic alcohol use, and nicotine dependence. Future studies are needed to shed light on how this knowledge can be used to improve the effects of digital SC programs for cancer survivors.

Trial Registration: Netherlands Trial register NTR6011/NL5434; https://onderzoekmetmensen.nl/nl/trial/22832

Introduction

In the past decade, digital interventions have commonly been used to target addictive behaviors, including smoking cessation (SC). Several systematic reviews have shown that these SC interventions can be effective, albeit with generally small effect sizes [ 1 - 4 ]. For example, the Cochrane review by Taylor et al [ 4 ] showed that the use of web-based SC interventions resulted in significantly higher rates of smoking abstinence compared to nonactive control groups, 6 months after randomization (risk ratio=1.15). Cancer survivors are a growing population who can benefit considerably from SC. Yet, the prevalence of people who smoke is about the same as in the general population, and research on effective digital SC interventions for cancer survivors is scarce [ 3 ]. Accordingly, not much is known about active ingredients or engagement factors of SC interventions targeting cancer survivors [ 5 ], despite engagement being an important moderator of the effect of digital SC interventions [ 6 ]. It is therefore important to look more closely into the predictors and moderators of engagement and outcome among this target group.

Although the primary effects of digital SC interventions are moderately positive on average, there is room for improvement. One possible explanation for the modest effects of digital SC interventions is the generally low adherence rates. Taylor et al [ 4 ] found that 18 out of 34 web-based SC studies had more than 50% attrition at follow-up. Analyzing whether the uptake of specific intervention components is related to better intervention outcomes increases the understanding of primary intervention effects [ 7 ]. Some studies on addictive behaviors investigated the relationship between intervention engagement and outcome [ 8 - 10 ]. A study by Perski et al [ 8 ] found that participants who completed more (varied) exercises had 64% higher odds of SC compared to participants who almost exclusively set an SC goal. Siemer et al [ 10 ], examining adherence to a blended SC intervention, revealed a dose-response relationship between the number of executed activities and smoking abstinence. Another study by Ramos et al [ 9 ] also found that intervention engagement, in terms of number of logins, forum visits, and number of participation badges, was a strong predictor of successful SC. Not all studies have shown that intervention engagement predicts intervention effectiveness, even contradictory effects are found. For example, Smith et al [ 11 ] showed that engagement with particular components of a digital SC intervention can be counterproductive when the content does not fit the participants’ needs.

Behavior change programs targeting SC notoriously encounter challenges when trying to reach target groups with the highest smoking rates (eg, groups with lower socioeconomic status [ 12 ] and groups with low literacy [ 13 ]). In addition, it could be useful for the improvement of intervention content and implementation to identify which characteristics predict engagement. This will help to improve the content and design of the intervention for the right target group [ 7 , 14 ]. Several studies have related digital SC intervention use to participant characteristics [ 8 , 15 - 17 ]. These studies showed that digital SC interventions were used longer or more frequently by older participants [ 8 , 15 ] and women [ 16 ]. Participants who had lower education, smoked more heavily, and had depressive symptoms were found to be less engaged with the digital SC intervention [ 17 ].

There is some evidence on the effects of digital SC interventions for cancer survivors. For example, a meta-analytic study by Mujcic et al [ 3 ] showed that digital and nondigital distance-based SC interventions for cancer survivors led to significantly reduced smoking rates compared to baseline (risk difference=0.29). However, research on the predictors and moderators of engagement and outcome of digital SC studies for cancer survivors is limited, while cancer survivors are a growing and diverse population [ 18 ]. A pilot study by Bricker et al [ 18 ] of an application on SC for cancer survivors showed greater acceptability, use, and effectiveness when compared to the national SC app for the general population.

In this study, we aim to investigate the predictors and moderators of engagement and outcome of a minimally guided digital intervention for cancer survivors called MyCourse-Quit Smoking (in Dutch “MijnKoers-Stoppen met Roken”) in a secondary analysis. The main effects study did not find a differential effect on SC between intervention and control at 6 months. In both groups, around a quarter abstained from smoking, and the number of cigarettes smoked was cut back by half [ 19 ]. With this secondary analysis, we aim to answer the following research questions: (1) Are participant characteristics related to intervention engagement at 6-month follow-up? (2) Is intervention engagement associated with tobacco use at 6-month follow-up? (3) Are participant characteristics related to tobacco use at 6-month follow-up?

For this paper, an exploratory secondary data analysis was carried out using data from a randomized controlled trial on the MyCourse-Quit Smoking digital intervention. The data used for this study were collected between November 2016 and September 2019.

Ethical Considerations

Ethics approval for the trial was acquired from an accredited medical research and ethics committee in The Netherlands (Toetsingscommissie Wetenschappelijk Onderzoek Rotterdam e.o. NL55921.101.16). Participants provided digital informed consent before inclusion in the trial [ 20 ]. Data were deidentified before processing or analysis. Identifying data were stored separately from research data. For each completed follow-up assessment, they were reimbursed €25 (approximately US $30).

A web-based screening questionnaire on the study website determined whether people were able to participate in the study. Eligible participants received an informed consent form via mail and had 30 days to sign the form. In the meantime, participants had the possibility to contact the research team or an independent physician for more information. After signing the informed consent form, they were asked to fill out a baseline questionnaire and were allocated to either the MyCourse group or the control group. Individuals in the control group were provided access to a noninteractive web-based informational brochure regarding the hazards associated with smoking and strategies for SC. The informational content encompassed both general SC information and content tailored to the unique needs of cancer survivors. Follow-up measurements were conducted at 3, 6, and 12 months after randomization. The study was conducted completely over the web, but after continued nonresponse, participants received a reminder by telephone. A more extensive description of the randomized controlled trial study procedures can be found elsewhere [ 20 ].

Participants

For the study, 165 adults who were diagnosed with any form of cancer in the past 10 years were recruited. Other eligibility criteria included having a PC or laptop in addition to an internet connection at home, having smoked 5 or more cigarettes per day in the past 7 days, having the intention and ability to participate in the 12-month study, and having the intention to quit smoking cigarettes. People who were pregnant; had insufficient mastery of the Dutch language; or self-reported suicidal ideation, dementia, severe depression, severe alcohol dependence, or acute psychosis were not eligible to participate in the study.

MyCourse Intervention

MyCourse-Quit Smoking is a digital minimally guided intervention that provides support for SC among cancer survivors. The intervention is based on empirically evaluated therapeutic approaches for SC in the general population: cognitive behavioral therapy [ 21 ], motivational interviewing [ 22 ], and acceptance and commitment therapy [ 23 ]. The intervention can be accessed via PC, tablet, and smartphone. At first login, participants receive instructions to set up a quit plan and gain access to 13 different exercises, information about smoking, quitting, and cancer, a web-based diary to track their tobacco use, and a peer support platform [ 20 ]. Exercises focused on different topics including previous experiences, high-risk situations, self-control measures, reinforcement, relapse prevention, and acceptance and commitment therapy. For the complete structure of the intervention, see Figure 3 in the protocol paper [ 20 ]. After the first login, all parts of the intervention could be accessed, and participants were free to choose how often and which parts of the intervention they wanted to use. Participants were only advised to use the intervention daily for 4 weeks.

The primary outcome measure in this study was the number of cigarettes smoked in the past 7 days at 6-month follow-up. Intervention engagement was measured using 3 indicators: the number of logins into the MyCourse-Quit Smoking intervention, the number of self-monitoring registrations of smoking urges and smoked cigarettes, and the number of completed intervention exercises. The following participant characteristics were extracted from the participant records: gender, age, educational level (higher or lower, where the minimum for the higher educational level was an academic university or university of applied sciences degree), and living situation (alone or together). We specifically looked at the presence of lung cancer and breast cancer (yes or no) among the participants in the analyses because lung cancer has a direct relationship with smoking and breast cancer was the most common type of cancer in the sample. In addition, patients with cancer at other sites were included in the analyses. Furthermore, the number of cancer sites (1 or >1) distinguished participants who reported that they had received multiple cancer diagnoses. The severity of nicotine dependence was measured by the 6-item Fagerström Test for Nicotine Dependence (FTND) [ 24 ]. Problematic alcohol use was measured using the Alcohol Use Disorders Identification Test (AUDIT) [ 25 ], a 10-item questionnaire on alcohol consumption patterns and problems experienced due to alcohol consumption. The AUDIT score was included as a variable because research has shown that people with a high risk of problematic alcohol use have a harder time quitting smoking and may benefit from different types of SC treatment [ 26 , 27 ]. The EQ-5D was used to measure the quality of life [ 28 ]. Comorbid anxiety, depression, and somatic symptoms were indicated using the Brief Symptom Inventory-18 questionnaire [ 29 ].

Statistical Analysis

Imputation of missing data.

Missing data for primary (ie, cigarettes smoked in the past 7 days) and secondary (ie, participant characteristics) outcome measures were multiple imputed (number of imputations=50) based on the intention-to-treat principle using the predictive mean matching method from the mice package in R (R Foundation for Statistical Computing) [ 29 ]. At the 6-month follow-up, the nonresponse rate (ie, participants who did not complete the 6-month questionnaire) was 27.7% (23/83) in the intervention group and 25.6% (21/82) in the control group. Data on intervention usage were not imputed. For the analyses containing engagement measures, participants who did not log in once were excluded.

Regression Analyses

Data were analyzed using R [ 30 ]. Bonferroni correction was applied in all analyses. The association between intervention engagement and participant characteristics within the intervention group was analyzed with a robust linear regression using the MASS package [ 31 ]. Whether participant characteristics and intervention engagement predicted intervention outcome within the intervention group was analyzed using robust linear mixed modeling (RLMM) with a random intercept using the robustlmm package [ 32 ]. RLMM is an effective analytical approach to account for outliers or skewed data [ 32 ]. The moderation analyses to investigate whether the study condition moderated the effect between participant characteristics and outcome were performed using RLMM with a random intercept and study condition × participant characteristics as the interaction term. This analysis is performed to assess whether the study condition (ie, being in the intervention group compared to the control group) moderates the association between participant characteristics and outcome. Model estimates, 95% CIs, and P values are reported. All base case analyses followed the intention-to-treat principle and used multiple imputed data sets. Sensitivity analyses were performed using observed data only.

Sample Characteristics

Participant characteristics are shown in Table 1 . The majority of participants were female (136/165, 82.4%), the mean age was 54.2 (SD 11.2) years, 29.1% (48/165) were living alone, and 41.2% (68/165) had completed higher education. On average, participants had smoked for 34.5 (SD 12.0) years and smoked 100 (SD 51.2) cigarettes per week. The main clinical effects of the MyCourse intervention and the results of the cost-effectiveness analysis can be found elsewhere [ 19 ].

Participant Characteristics and Intervention Engagement

Of all 83 participants of the intervention group, 56 people logged into the MyCourse portal at least once and thus were included in the analysis. When comparing the 56 people who logged in at least once with the 27 people who did not log in once at all baseline characteristics mentioned in Table 1 , only the number of patients with uterine cancer differed significantly between the 2 groups ( P <.05), with 5 patients with uterine cancer who did not log in once and 2 patients with uterine cancer that logged in at least once. In total, 82 participants in the control group were not included in the analysis. Among the 56 MyCourse users, the average number of logins was 21 (SD 41.0; median 5.5, IQR 3-18.5), the average amount of self-monitoring registrations was 31 (SD 53.9; median 5, IQR 2-22), and the average amount of completed exercises was 6.5 (SD 5.1; median 4, IQR 2-12). As shown in Table 2 , female participants showed a significantly lower number of logins in the MyCourse-Quit Smoking intervention than male participants ( P =.004). The relationship between sex and other indicators of intervention engagement was nonsignificant. Furthermore, a higher AUDIT score at baseline was associated with a significantly higher number of logins ( P <.001) and diary registrations ( P <.001) but not with the number of completed exercises ( P =.05).

a A Bonferroni correction was applied based on 11 tests resulting in an α of .0045.

b Female participants showed a significantly lower number of logins in the MyCourse-Quit Smoking intervention than male participants.

c FTND: Fagerström Test for Nicotine Dependence.

d BSI-18: Brief Symptom Inventory-18.

e AUDIT: Alcohol Use Disorders Identification Test.

f A higher AUDIT score at baseline was associated with a significantly higher number of logins and diary registrations but not with the number of completed exercises.

Intervention Engagement, Participant Characteristics, and Smoking Behavior

Table 3 shows the outcomes of the analysis on the association between intervention engagement and smoking behavior among the 56 participants who logged in to the MyCourse portal at least once. No significant effects were found between intervention engagement and smoking behavior. Table 3 also shows the association between several participant characteristics and smoking behavior among the 83 participants of the intervention group. The results show that a higher FTND score at baseline is associated with a significantly greater reduction of the 7-day sum of smoked cigarettes after 6 months in the intervention group ( P =.002). None of the other participant characteristics or measures of engagement predicted smoking behavior at 6 months.

a A Bonferroni correction was applied based on 14 tests resulting in an α of .004.

b FTND: Fagerström Test for Nicotine Dependence.

c A higher FTND score at baseline is associated with a significantly greater reduction of the 7-day sum of smoked cigarettes after 6 months in the intervention group.

Moderation Analysis

Table 4 reports the outcomes of the moderation analysis on the interaction effect of participant characteristics and study condition on the number of cigarettes smoked in the past 7 days among the 165 participants at 6-month follow-up. No significant effects were found in this analysis.

c BSI-18: Brief Symptom Inventory-18.

d AUDIT: Alcohol Use Disorders Identification Test.

Sensitivity Analysis

The association between sex and the number of logins on the nonimputed data did not reach significance after the Bonferroni correction ( P =.006). The association between the FTND and 7-day cigarette smoking at 6-month follow-up in the nonimputed data did not reach significance after Bonferroni correction ( P =.10). In the moderation analysis, after Bonferroni correction on the nonimputed data, the interaction effect of the number of cancer sites and study condition on smoking behavior reached significance ( P =.002). For all other analyses, the results did not change significantly. Detailed results of the sensitivity analyses can be found in Multimedia Appendix 1 .

Principal Findings

In this study, we evaluated hypothesized predictors and moderators of intervention engagement and smoking behavior in MyCourse-Quit Smoking, a digital SC intervention for cancer survivors. With regard to the relationship between participant characteristics and intervention engagement, it was found that female participants logged on significantly less often than male participants. This effect should nevertheless be interpreted with caution since the number of male participants in the sample was low (n=8). Moreover, previous research shows that female participants are generally more engaged in digital SC interventions than male participants [ 16 , 33 - 37 ]. A significant positive association between the baseline AUDIT score and intervention engagement was found; a higher AUDIT score at baseline was related to a higher number of logins and diary registrations in the MyCourse intervention. There was no effect of the baseline AUDIT score on the number of completed exercises. Previous studies showed that participants with a higher risk of alcohol dependence had a harder time to quit smoking, and therefore needed more support from the intervention, as demonstrated in several previous studies [ 26 , 27 ]. Toll et al [ 27 ] showed that people who drink more heavily were less likely to quit smoking, but problematic alcohol use was not measured. Sells et al [ 26 ] pointed out that people with a high risk of problematic alcohol use may need more intensive intervention in order to quit smoking, whereas people with a high risk of problematic alcohol use were defined with an AUDIT score higher than 7. However, in this study, we did not find an effect of the AUDIT score on smoking behavior. Furthermore, participants of the MyCourse-Quit Smoking trial had generally low AUDIT scores (mean 3.6, SD 4.7), and few participants with a score higher than 7 (21/165).

Regarding the association between participant characteristics and smoking behavior, we found that participants of the MyCourse intervention who had higher nicotine dependence scores at baseline showed a greater reduction in the number of smoked cigarettes in the past 7 days at the 6-month follow-up. This negative association between nicotine dependence at baseline and tobacco use at follow-up is a reasonable finding because it is likely that less addicted participants at baseline already smoke fewer cigarettes than highly addicted participants, and therefore, a smaller reduction of cigarettes at 6 months is possible. This finding does not indicate whether heavier nicotine dependence predicts SC, as participants can greatly reduce the number of smoked cigarettes but not enough to completely quit smoking. Previous research shows that, in general, less severe nicotine dependence is associated with a higher SC rate [ 38 , 39 ].

The analyses on the association between intervention engagement (ie, the number of logins, self-monitoring registrations, and exercises) and the outcome did not yield any significant effects. This study showed the overall prevailing pattern of the majority of participants quitting the use of the intervention in the first few days and a smaller group that uses the intervention for a longer period [ 40 ]. However, other studies on digital SC interventions have shown a dose-response relationship between intervention engagement and outcome [ 9 , 10 , 41 ], with higher engagement predicting greater SC rates, although this is sometimes limited to certain engagement measures [ 34 ] or with low quality of the evidence due to low follow-up rates [ 9 ]. For example, Heminger et al [ 34 ] did not find a significant association between program dose and SC, but the use of specific intervention elements (eg, making a pledge toward a smoke-free life and tracking saved money and health benefits gained after quitting) was associated with SC. For future research, it is therefore important to properly define engagement, differentiate between indicators of engagement, and use empirically effective intervention techniques in order to enhance engagement [ 6 ].

The moderation analysis did not yield any significant effects. This indicates that being in the intervention group, compared to the control group, does not amplify the effect between any of the participant’s characteristics and tobacco use, and hence no specific participant characteristic renders participants more or less likely to be successful when participating in the MyCourse intervention.

Limitations

The initial study was 80% powered to detect a relative risk of 2.1 in SC [ 20 ], while this explored different outcome variables, potential moderator effects herein, and made comparisons other than between treatment arms. Hence, the initial sample size calculation might not be applicable. Post hoc power analyses were omitted, as these would merely reflect the already obtained P value [ 42 ]. While the applied Bonferroni correction accounted for multiple comparisons, it might be overly strict in our case [ 43 ]. Furthermore, the tendency to overfit data might also be a problem for linear mixed modeling analyses. The study had missing data, which might have caused bias in the results. On the other hand, as a strength of this study, multiple imputation was applied to compensate for the missing values, and the sensitivity analysis did not reveal any substantial differences in the analyses without imputation. Another limitation is the sample size of the analyses for the first research question, especially for the subgroup analyses of sex and living situation. Since some of the categories of these variables had small group sizes, the outcomes of the analyses should be interpreted with caution.

Clinical Implications

The MyCourse intervention is presumably more engaging for people who smoke and people with moderate to high alcohol dependence. Furthermore, this study did not identify any specific subgroups where the MyCourse-Quit Smoking intervention might be particularly effective or ineffective.

Conclusions

This study aimed to provide more insight into predictors and moderators of engagement and outcome for a digital SC intervention targeting cancer survivors. Overall, a limited number of associations was found between participant characteristics, engagement, and smoking behavior. Female participants accessed the intervention less often than male participants, and participants with higher AUDIT scores accessed the intervention more often and had more diary registrations than participants with lower AUDIT scores. Greater nicotine dependence at baseline was associated with a greater reduction in number of cigarettes at 6 months. Future studies in a larger sample and with a preregistered analysis plan are needed to corroborate these findings and shed light on how this knowledge can be used to improve the effects of digital SC programs.

Acknowledgments

This study was funded by the Dutch Cancer Institute (KWF kankerbestrijding; grant TBOS2014–7169). The sponsors did not have influence on the design, data collection, analysis, and interpretation of the data, nor in writing the paper or the decision to submit it for publication. The authors would like to thank Diede Kesler for her assistance in the recruitment and follow-up assessments of the participants and Yvonne Borghans for her assistance in the follow-up assessments of the participants.

Authors' Contributions

AM, MvL, BB, RE, and MB contributed to the conception and data collection of the original research. RA, AM, WdH, and MB conceived the research questions and design for this study. ML, BB, and RE commented on or rewrote the design and research questions. RA and WdH performed the statistical analyses. RA wrote the first draft of the manuscript. AM, WdH, MvL, BB, RE, and MB commented on the draft and rewrote sections of the draft. All authors approved the final version of the manuscript.

Conflicts of Interest

None declared.

Sensitivity analysis of all performed analyses.

• Do HP, Tran BX, Le Pham Q, Nguyen LH, Tran TT, Latkin CA, et al. Which eHealth interventions are most effective for smoking cessation? A systematic review. Patient Prefer Adherence. 2018;12:2065-2084. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Graham AL, Carpenter KM, Cha S, Cole S, Jacobs MA, Raskob M, et al. Systematic review and meta-analysis of internet interventions for smoking cessation among adults. Subst Abuse Rehabil. 2016;7:55-69. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mujcic A, Blankers M, Bommelé J, Boon B, Berman AH, Verdonck-de Leeuw IM, et al. The effectiveness of distance-based interventions for smoking cessation and alcohol moderation among cancer survivors: a meta-analysis. Psychooncology. 2020;29(1):49-60. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Taylor GMJ, Dalili MN, Semwal M, Civljak M, Sheikh A, Car J. Internet-based interventions for smoking cessation. Cochrane Database Syst Rev. 2017;9(9):CD007078. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sheeran P, Jones K, Avishai A, Symes YR, Abraham C, Miles E, et al. What works in smoking cessation interventions for cancer survivors? A meta-analysis. Heal Psychol. 2019;38(10):855-865. [ CrossRef ]
• Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. 2017;7(2):254-267. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sieverink F, Kelders S, Poel M, van Gemert-Pijnen L. Opening the black box of electronic health: collecting, analyzing, and interpreting log data. JMIR Res Protoc. 2017;6(8):e156. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Perski O, Watson NL, Mull KE, Bricker JB. Identifying content-based engagement patterns in a smoking cessation website and associations with user characteristics and cessation outcomes: a sequence and cluster analysis. Nicotine Tob Res. 2021;23(7):1103-1112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ramos LA, Blankers M, van Wingen G, de Bruijn T, Pauws SC, Goudriaan AE. Predicting success of a digital self-help intervention for alcohol and substance use with machine learning. Front Psychol. 2021;12:734633. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Siemer L, Brusse-Keizer MG, Postel MG, Allouch SB, Bougioukas AP, Sanderman R, et al. Blended smoking cessation treatment: exploring measurement, levels, and predictors of adherence. J Med Internet Res. 2018;20(8):e246. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Smith W, Ploderer B, Wadley G, Webber S, Borland R. Trajectories of engagement and disengagement with a story-based smoking cessation app. New York, NY, United States. Association for Computing Machinery; 2017. Presented at: CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems; May 6-11, 2017:3045-3056; Denver Colorado USA. URL: https://dl.acm.org/doi/proceedings/10.1145/3025453 [ CrossRef ]
• van Wijk EC, Landais LL, Harting J. Understanding the multitude of barriers that prevent smokers in lower socioeconomic groups from accessing smoking cessation support: a literature review. Prev Med. 2019;123:143-151. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nouri SS, Avila-Garcia P, Cemballi AG, Sarkar U, Aguilera A, Lyles CR. Assessing mobile phone digital literacy and engagement in user-centered design in a diverse, safety-net population: mixed methods study. JMIR Mhealth Uhealth. 2019;7(8):e14250. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Beintner I, Görlich D, Berger T, Ebert DD, Zeiler M, Camarano RH, et al. Interrelations between participant and intervention characteristics, process variables and outcomes in online interventions: a protocol for overarching analyses within and across seven clinical trials in ICare. Internet Interv. 2019;16:86-97. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bricker JB, Sridharan V, Zhu Y, Mull KE, Heffner JL, Watson NL, et al. Trajectories of 12-month usage patterns for two smoking cessation websites: exploring how users engage over time. J Med Internet Res. 2018;20(4):e10143. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Iacoviello BM, Steinerman JR, Klein DB, Silver TL, Berger AG, Luo SX, et al. Clickotine, a personalized smartphone app for smoking cessation: initial evaluation. JMIR Mhealth Uhealth. 2017;5(4):e56. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zeng EY, Vilardaga R, Heffner JL, Mull KE, Bricker JB. Predictors of utilization of a novel smoking cessation smartphone app. Telemed J E Health. 2015;21(12):998-1004. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bricker JB, Watson NL, Heffner JL, Sullivan B, Mull K, Kwon D, et al. A smartphone app designed to help cancer patients stop smoking: results from a pilot randomized trial on feasibility, acceptability, and effectiveness. JMIR Form Res. 2020;4(1):e16652. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mujcic A, Blankers M, Boon B, Verdonck-de Leeuw IM, Smit F, van Laar M, et al. Effectiveness, cost-effectiveness, and cost-utility of a digital smoking cessation intervention for cancer survivors: health economic evaluation and outcomes of a pragmatic randomized controlled trial. J Med Internet Res. 2022;24(3):e27588. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mujcic A, Blankers M, Boon B, Engels R, van Laar M. Internet-based self-help smoking cessation and alcohol moderation interventions for cancer survivors: a study protocol of two RCTs. BMC Cancer. 2018;18(1):364. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Perkins KA, Conklin CA, Levine MD. Cognitive-Behavioral Therapy for Smoking Cessation: A Practical Guidebook to the Most Effective Treatments. Milton Park, UK. Routledge; 2013.
• Lindson N, Thompson TP, Ferrey A, Lambert JD, Aveyard P. Motivational interviewing for smoking cessation. Cochrane Database Syst Rev. 2019;7(7):CD006936. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McCallion EA, Zvolensky MJ. Acceptance and Commitment Therapy (ACT) for smoking cessation: a synthesis. Curr Opin Psychol. 2015;2:47-51. [ CrossRef ]
• Heatherton TF, Kozlowski LT, Frecker RC, Fagerström KO. The Fagerström Test for Nicotine Dependence: a revision of the Fagerström tolerance questionnaire. Br J Addict. 1991;86(9):1119-1127. [ CrossRef ] [ Medline ]
• Saunders JB, Aasland OG, Babor TF, de la Fuente JR, Grant M. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption--II. Addiction. 1993;88(6):791-804. [ CrossRef ] [ Medline ]
• Sells JR, Waters AJ, MacLean RR. Evaluating the influence of at-risk alcohol use on factors associated with smoking cessation: combining laboratory and ecological momentary assessment. Drug Alcohol Depend. 2017;179:267-270. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Toll BA, Martino S, O'Malley SS, Fucito LM, McKee SA, Kahler CW, et al. A randomized trial for hazardous drinking and smoking cessation for callers to a quitline. J Consult Clin Psychol. 2015;83(3):445-454. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rabin R, de Charro F. EQ-5D: a measure of health status from the EuroQol Group. Ann Med. 2001;33(5):337-343. [ CrossRef ] [ Medline ]
• Derogatis LR, Fitzpatrick M. The SCL-90-R, the Brief Symptom Inventory (BSI), and the BSI-18. In: Maruish ME, editor. The use of Psychological Testing for Treatment Planning and Outcomes Assessment: Instruments for Adults, Vol 3, 3rd Edition. Mahwah, NJ, US. Lawrence Erlbaum Associates Publishers; 2004:1-41.
• A language and environment for statistical computing (version 3.0.1). R Foundation for Statistical Computing. Austria.; 2022. URL: http://www.r-project.org [accessed 2024-04-03]
• Ripley B, Venables B, Bates DM, Hornik K, Gebhardt A, Firth D, et al. Package 'mass'. Cran r. 2013;538:113-120. [ FREE Full text ]
• Koller M. Robustlmm: an R package for robust estimation of linear mixed-effects models. J Stat Soft. 2016;75(6):1-24. [ CrossRef ]
• Balmford J, Borland R, Benda P. Patterns of use of an automated interactive personalized coaching program for smoking cessation. J Med Internet Res. 2008;10(5):e54. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Heminger CL, Boal AL, Zumer M, Abroms LC. Text2Quit: an analysis of participant engagement in the mobile smoking cessation program. Am J Drug Alcohol Abuse. 2016;42(4):450-458. [ CrossRef ] [ Medline ]
• Nash CM, Vickerman KA, Kellogg ES, Zbikowski SM. Utilization of a web-based vs integrated phone/web cessation program among 140,000 tobacco users: an evaluation across 10 free state quitlines. J Med Internet Res. 2015;17(2):e36. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Strecher VJ, McClure J, Alexander G, Chakraborty B, Nair V, Konkel J, et al. The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial. J Med Internet Res. 2008;10(5):e36. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wangberg SC, Bergmo TS, Johnsen JAK. Adherence in internet-based interventions. Patient Prefer Adherence. 2008;2:57-65. [ FREE Full text ] [ Medline ]
• Lindberg A, Niska B, Stridsman C, Eklund BM, Eriksson B, Hedman L. Low nicotine dependence and high self-efficacy can predict smoking cessation independent of the presence of chronic obstructive pulmonary disease: a three year follow up of a population-based study. Tob Induc Dis. 2015;13(1):27. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rojewski AM, Tanner NT, Dai L, Ravenel JG, Gebregziabher M, Silvestri GA, et al. Tobacco dependence predicts higher lung cancer and mortality rates and lower rates of smoking cessation in the national lung screening trial. Chest. 2018;154(1):110-118. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chen J, Houston TK, Faro JM, Nagawa CS, Orvek EA, Blok AC, et al. Evaluating the use of a recommender system for selecting optimal messages for smoking cessation: patterns and effects of user-system engagement. BMC Public Health. 2021;21(1):1749. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Perski O, Crane D, Beard E, Brown J. Does the addition of a supportive chatbot promote user engagement with a smoking cessation app? An experimental study. Digit Health. 2019;5:2055207619880676. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lakens D. Sample size justification. Collabra Psychol. 2022;8(1):33267. [ FREE Full text ] [ CrossRef ]
• VanderWeele TJ, Mathur MB. Some desirable properties of the Bonferroni correction: is the Bonferroni correction really so bad? Am J Epidemiol. 2019;188(3):617-618. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by J Bender; submitted 07.02.23; peer-reviewed by K Pebley; comments to author 03.10.23; revised version received 26.01.24; accepted 25.02.24; published 20.06.24.

©Rosa Andree, Ajla Mujcic, Wouter den Hollander, Margriet van Laar, Brigitte Boon, Rutger Engels, Matthijs Blankers. Originally published in JMIR Cancer (https://cancer.jmir.org), 20.06.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on https://cancer.jmir.org/, as well as this copyright and license information must be included.