Why markov model

The use of the willingness to pay thresholds is further discussed in section 4. The objective of this paper is to show the characteristics of pharmacoeconomic studies and models, with a particular focus on Markov and Semi-Markov models. Using a cost utility model for a vaccination program for Dengue fever, we show how these kinds of models work and what are their limits. We also raise the attention on incorrect hypothesis specification and incorrect interpretation of the outcomes.

Moreover, we suggest the use of some tools to bypass the limits of this approach and give some recommendations for further research. The remainder of the paper is structured as follows. Section 2 explains the theoretical background of pharmacoeconomic models and their critiques in literature. Section 3 exploits a Semi Markov model cost utility analysis for a vaccination program, which serves as background for a more accurate discussion in section 4. Section 5 ends the paper with some concluding remarks.

In literature, there are three main types of statistical approaches to the implementation of pharmacoeconomic studies 4 : regression models, decision trees, and Markov chain models. This paper will primarily focus on the latter.

The most important advantage of regression models is their capability to use least-squares linear regression techniques to explore the marginal impact of covariates on incremental cost-effectiveness instead of the usual models that aggregate cost and effect differences 5.

Linear regression has been primarily used by various researchers to compute the net monetary benefit NMB and the net health benefit NHM 6 — 9 but it has also been used as an epidemiological model by interpreting the estimated coefficients as the risk factor weights 4.

Using these models, it is straightforward to increase the number of explanatory variables in order to examine their influence on cost-effectiveness directly. Decision trees are among the simplest model used in pharmacoeconomics. They are simple directed graphs without recursion Figure 2 and represent a sequence of chance events and decisions overtime 10 , Decision trees have been used for many health care problems: Caekelbergh et al.

Simple decision trees usually follow the same paradigm: a the decision node; b the decision strategy; c the outcome nodes. There is no decision if no value is assigned to the outcomes. Both input probabilities and values in decision trees are generally obtained from literature, guidelines and experts 4. Markov models were first developed by the Russian scientist Andrei Markov — They are represented as partially cyclic directed graphs Figure 3. In the field of pharmacoeconomic analysis, they are exceptionally suited for diseases that involve an ongoing over time risk e.

The ongoing risk leads to important consequences: first, the time when events occur is unknown; second, an event can happen more than once, thus it is difficult in this case to use a decision tree In pharmacoeconomic Markov models, health statuses are represented as Markov states, and the health changes as transition probabilities between states. Although continuous time Markov models can be built [e.

Although modeling continuous time is somehow required, discrete event simulation DES models are often suggested. DES is a flexible modeling method where entities may interact with each other for resources in a system, and each interaction between entities is an event A DES approach, in contrast to Markov modeling, offers some advantages: retention of patient history; risk profile update after each event and time flexibility.

In view of that, although in Markov models the length of a cycle is fixed, in DES the simulation time can adapt directly to the time the next event occurs Markov models compensate their limitations with their simplicity, as they can be visually inspected for programming errors, and can be tested straightforwardly for technical replication Notationally, a Markov chain is a sequence of random variables X 1 , X 2 , This means that, for all n and states x n.

This set is defined as the state space of the chain. Transition probabilities are defined as. Usually, Markov models are time-homogenous. This means that there are no changes in the transition probabilities as time goes on, but in modeling health care generally non-homogenous Markov models also called Semi-Markov models are used In this case, the transition probabilities depend upon the amount of time that has passed.

In both cases homogeneous or non-homogeneous models the process may, or may not, be stationary. A process is said to be stationary if it is invariant under an arbitrary shift of the time origin A discrete time Markov model consists of one or more communicating classes that form a set of states that communicate.

If the chain is composed of one communicating class only, the chain is said to be irreducible In order to use a Markov model in a pharmacoeconomic study, it is fundamental to attach weights to the states, that allow the analyst to estimate cost and health outcomes.

As an example, for predicting life expectancy a zero weight is attached to the death state and a unit weight is attached to the other states. Running the model for many cycles provides an estimate of the average life expectancy. In the case of an economic evaluation, researchers are interested in the quality adjusted life year, or in the effect of a therapy. Thus, these kinds of elements need to be attached in a similar way to the life-expectancy case.

On the cost side, the model behaves likely: the costs of spending one cycle in each of the states are assigned to that state and, as the model runs for many cycles, the total cost is obtained by summing across those cycles Moreover, to make this economic model more realistic, adjustments for differential time are needed. These adjustments are done by discounting outcomes and costs. This allows the user to compare costs and outcomes in terms of a net present value.

In the pharmacoeconomic literature, the use of Markov models or Semi Markov models is vast and growing. Anis et al. Leelahavarong et al. Although Markov models can be used for infectious disease modeling, dynamic models are better suited for the task. For instance, DePasse et al. Moreover, the two approaches can be used simultaneously.

As an example, in Khazeni et al. Furthermore, Yaesoubi and Cohen 25 and Haeussler et al. With this approach, in contrast to the deterministic compartmental models, it is possible to approximate the spread of the disease in large populations with a small state-space size, controlling for both an acceptable degree of accuracy and computational time.

Although Markov models sometimes can be applied for infectious disease modeling, they are also suitable for modeling pharmacoeconomics for transplant, as in Jensen et al. Both studies conducted a CUA for kidney transplant compared to dialysis.

An intensive use of Markov models can also be observed in pharmacogenomics testing and precision medicine: by doing a genetic test to a patient, it is possible to formulate a personalized and more successful therapy. Therefore, in theory the cost of the test should be counterbalanced by the higher effects of the therapy. Examples of this kind of models are genetic testing for major depressive disorder 29 , cardiovascular prevention 30 , and epilepsy Moreover, Markov Models for economic evaluation of healthcare interventions are particularly suitable for modeling the cost effectiveness of new health care interventions for non-communicable parasitic diseases.

These diseases are such that the parasite life cycle does not include direct host-to-host transmission, thus they are not contagious directly among humans Studies on intervention for these diseases are those of Shankar et al.

Finally, Seo et al. In the framework of Dengue fever, many papers using Markov models have been proposed in the literature for economic evaluation of healthcare interventions. Valuable studies are that of Perera et al.

Beside Markov models, which in their standard version are static models and cannot capture indirect effects such herd immunity or increase transmissibility, recent economic evaluations of interventions on Dengue fever often use dynamic transmission models, described for instance in Flasche et al.

Moreover, Lee et al. In this section, we show some examples to demonstrate how and why Markov models are often used in pharmacoeconomic analysis. Our analyses are carried out using the R software for statistical computing 46 , and in particular, the package Heemod Since our main aim is to show how Markov models work in Pharmacoeconomics, we exploit a semi-Markov model on the cost utility of a vaccine for the Dengue fever disease.

This is a simplified version of the model described in 36 which analyzes, from a societal perspective, a Dengue CYD-TDV vaccination program following a pre-vaccination serological screening in Sri Lanka.

Since the main purpose of our analyses is showing pros and cons of semi-Markov models in Pharmacoeconomics, we do not consider the part of the model involving screening. Our model, moreover, does not take into account the herd effect resulted from the dengue vaccination campaign nor the vaccine efficacy waning, as well as it does not deal with the well-known CYT-TDV safety issues 48 , as we do not consider the serological screening as in Perera et al.

However, these limitations do not undermine the relevance of our study. The semi-Markov model we use for Dengue is just an instrument to emphasize advantages and disadvantages of Markov models in Pharmacoeconomics and to highlight the critical points that need to be overlooked to pursue rational health care policy decisions.

Cost, utilities, and transition probabilities of Dengue fever disease, retrieved from literature, are reported in Table 1 with the related references in the last column. The probability of showing the Dengue symptoms is correlated with the age of the cohort and thus a semi-Markov model is considered. A graphical representation of the model is shown in Figure 4.

Costs and utilities, in form of QALYs, are attached to each of the Markov states, and a discount rate is applied for both benefits and costs.

A time horizon of 10 years is used for the cost utility analysis. Two identical cohorts of 1, 9-years-old children compose both the intervention and control groups. Early in the disease the entire population is in a healthy state, but each year corresponding to a Markov cycle there is a chance to get sick and to show Dengue fever symptoms, which require moderate medical attention.

Subjects with Dengue fever can either get cured, and go back in the healthy state, or worse their condition and show a Dengue hemorrhagic fever DHF , which requires intensive care. DHF patients have a certain probability to die, or they can recover and go back to the healthy state. The probability to show symptoms changes according to the age of the cohort. Examples of health states that might be included in a simple Markov model for a cancer intervention are: progression-free, post-progression and dead.

Time spent in each disease state for a single model cycle and transitions between states is associated with a cost and a health outcome. Costs and health outcomes are aggregated for a modelled cohort of patients over successive cycles to provide a summary of the cohort experience, which can be compared with the aggregate experience of a similar cohort, for example one receiving a different comparator intervention for the same condition.

One way to simulate this weather would be to just say "Half of the days are rainy. Therefore, every day in our simulation will have a fifty percent chance of rain. Did you notice how the above sequence doesn't look quite like the original? The second sequence seems to jump around, while the first one the real data seems to have a "stickyness". In the real data, if it's sunny S one day, then the next day is also much more likely to be sunny.

We can minic this "stickyness" with a two-state Markov chain. When the Markov chain is in state "R", it has a 0. Likewise, "S" state has 0. Finite Markov chains. New York: Springer; Google Scholar.

Iosifescu M. Finite Markov processes and their applications. Mineola, NY: Dover; Taylor HM, Karlin S. An introduction to stochastic modeling. Boston: Academic Press; Robine J-M, Ritchie K. Healthy life expectancy: evaluation of global indicator of change in population health. Br Med J. CAS Google Scholar. A multistate analysis of active life expectancy.

Public Health Rep. Smoking, physical activity, and active life expectancy. Am J Epidemiol. Socioeconomic inequalities in life and health expectancies around official retirement age in 10 Western-European countries.

J Epidemiol Community Health. A cardiovascular life history. A life course analysis of the original Framingham Heart Study cohort. Eur Heart J. A continuous time Markov model for the length of stay of elderly people in institutional long-term care. Estimating stroke-free and total life expectancy in the presence of non-ignorable missing values. Zimmer Z, Rubin S. Life expectancy with and without pain in the US, elderly population.

J Gerontol A. Nusselder WJ, Peeters A. Successful aging: measuring the years lived with functional loss. A healthy bottom line: healthy life expectancy as an outcome measure for health improvement efforts. Milbank Q. Transitions between states of disability and independence among older persons. PubMed Google Scholar.

Transitions between frailty states among community-living older persons. Ann Intern Med. New methods for analyzing active life expectancy. J Aging Health. Counting labeled transitions in continuous-time Markov models of evolution. J Math Biol. Lifetime reproduction and the second demographic transition: stochasticity and individual variation. Demogr Res. Caswell H, Zarulli V. Matrix methods in health demography: a new approach to the stochastic analysis of healthy longevity and DALYs.

Popul Health Metrics. Statistical models based on counting processes, 2nd edn. Howard RA. Dynamic programming and Markov processes. Caswell H. Beyond R0: demographic models for variability of lifetime reproductive output. Van Daalen S, Caswell H. Lifetime reproductive output: individual stochasticity, variance, and sensitivity analysis. Theor Ecol. Statistical inference about Markov chains.

Ann Math Stat. Bootstrapping a finite state Markov chain. Estimation of the transition matrix of a discrete-time Markov chain. Health Econ. Methods and applications. Matrix population models. Sutradhar R, Cook RJ. Analysis of interval-censored data from clustered multistate processes: application to joint damage in psoriatic arthritis. Global, regional, and national life expectancy, all-cause mortality, and cause-specific mortality for causes of death, — a systematic analysis for the Global Burden of Disease Study Long-term survival and cause-specific mortality in patients with cirrhosis of the liver: a nationwide cohort study in Denmark.

J Clin Epidemiol. Harvald B, Madsen S. Long-term treatment of cirrhosis of the liver with prednisone. J Intern Med. The mstate package for estimation and prediction in non- and semi-parametric multi-state and competing risk models. Comput Methods Prog Biomed. J Stat Softw. Rogers A, Ledent J. Increment-decrement life tables: a comment. Change in disability-free life expectancy for Americans 70 years old and older.

Disability in activities of daily living: patterns of change and a hierarchy of disability. Am J Public Health. Recovery from disability among community-dwelling older persons. J Am Med Assoc. Patterns of functional decline at the end of life.