Brian Mittman: Okay, so that recording message is my cue that we are ready to go so welcome back everyone, my apologies for missing of the prior session, when I was.
Brian Mittman: enjoying visiting with grandkids and our daughter's graduation but i'm happy to be here and to have the honor of moderating discussion with our two distinguished guests couple of points before we launch into panel discussion that is.
Brian Mittman: To ensure that you are aware of and in keeping up with the various deadlines to complete your materials for your final presentations and submit those on time, if you expect to.
Brian Mittman: miss the deadline for any reason, please let us know in advance and then ask about some alternatives, but dumb, otherwise the expectation is that everything will come in.
Brian Mittman: per the deadlines and then the other point is to encourage you to Of course some join both of the.
Brian Mittman: sessions, so that you benefit not only from hearing others who present in the session where you're presenting, but the second half of the program as well, I think those capstone presentations are always.
Brian Mittman: Quite insightful and helpful in understanding the full breadth of issues and, in many cases they're similar projects and hearing the different ways in which you know those projects have been designed is especially useful Erica any other comments.
Brian Mittman: encouragement for the capstones.
Erica Breslau: i'm hearing wonderful sorry feedback from the small group leaders who have been some the peer reviewers.
Erica Breslau: that people are really enjoying reviewing each other's projects so i'm feeling very heartened by that and Brian I wasn't listening, but did you mention that the.
Erica Breslau: link to present your to submit all of your presentations is here in the chat so please copy it and that's about all so I guess, we can start with do you want to introduce our two colleagues.
Brian Mittman: Sure, so let me start with the colleague who i've known for the longest and, as with Norton, and I say how long but it's been one of the earlier.
Brian Mittman: thinkers and contributors to implementation, science, even going back to her doctoral program and I had the opportunity to work with when on some.
Brian Mittman: Issues of scale up in spread in implementation signs, but had her contributions as an academic and that alone in the small but mighty implementation science team within.
Brian Mittman: nci and for those of you who are multi level interventionist without necessarily a focus on implementations signs and you'll be mentioned this before that's the team that you should be paying attention to join you the email, because many of the workshops that they.
Brian Mittman: sponsor conduct implementation science in fact art, more broadly, relevant and then turning to Michael helper and i've known not quite as long but.
Brian Mittman: One of the again I guess small but mighty is a term that i'll use in terms of health economics within the field.
Brian Mittman: You know it's a set of issues that many of us are not completely comfortable with, but I can tell you is embedded researcher both within va now, especially within Kaiser that doesn't have the benefit of.
Brian Mittman: Taxpayers in Congress to bail us out when needed, being aware of the economic issues, the costs and even, as you know, I hope, corey has now seen the light and according to point out.
Brian Mittman: The legislation that reauthorized because it actually encourages us to think about costs issues, whereas That was a four letter word during the first round, so when you know, the idea that we need more and better attention to economic issues.
Brian Mittman: And certainly the case in multi level intervention research is, it is an implementation science so it's nice to have.
Brian Mittman: You know, an expert in that area, so let me begin with some questions in when tag you first.
Brian Mittman: and ask if you could talk a little bit about how we define an evidence based intervention in the issue of how much evidence of effectiveness is needed.
Brian Mittman: From our multi level of intervention studies and others before we begin to work on the implementation issues.
Brian Mittman: And do so in a way that doesn't put the cart before the horse move too far, but also doesn't put us on a 20 or 30 year journey, where you know work policy and practice relevance is lost.
Wynne Norton: yeah that's great Brian and just before I try to answer that question just want to say thank you to everyone for being here today, and thank you particularly to Erica and Brian I think it's almost been 15 years I was just doing the calculations.
Wynne Norton: But, and then, ironically, you and I worked on a scale been spread say they are at conferences it's nice to kind of circle back here and i'm happy to share actually that report, there are some there's a couple nice picture, Brian.
Wynne Norton: it's it's it's classic and Erica Thank you so much you've been a fantastic colleague during my time here and ci and it's very to be part of this.
Wynne Norton: Training program and to speak with all of you today and it's really a fantastic program so i'm very excited that everyone has been able to participate.
Wynne Norton: Now so it's a it's a may switch here to answering the question it's a fantastic question it's, one that is asked all the time, at least with an implementation science and i'm sure within multi level intervention research as well, I think you know there isn't any.
Wynne Norton: hard line, so to say, so to speak, of what is sufficient versus what is not sufficient it's kind of there's some variability I think it's a function of what intervention you're talking about.
Wynne Norton: The potential for and kind of the quality and quantity of evidence for in supportive that intervention.
Wynne Norton: You know it's in discussion among the research community and practice Community currently I think in terms of you know, since i'm at a funding agency, what we see currently is that reviewers.
Wynne Norton: And us funding studies, like to see enough to justify that this is worthy of scale been spread on a very broad level, of course, so.
Wynne Norton: Whether that's you know one rct five RC cheese, you know, a mix of you know, qualitative data and supported this intervention plus you know some type of quasi experimental design that's kind of up in the air it's not.
Wynne Norton: As as kind of narrow as let's say FDA needs criteria for what is is worthy of putting on the market or usps tf ratings of a or above which really indicates that it should be widely used so it's it's you know, a kind of a it depends, but.
Wynne Norton: I would say, I think one of the questions that we don't often ask is if we don't have it and oftentimes it's.
Wynne Norton: You know it's better to put something out there if we don't have you know, we have some evidence, but maybe not enough.
Wynne Norton: Based on you know FDA or usps g of tired type of ratings I think i'm and would suggest that folks think a little bit more about.
Wynne Norton: If we don't have enough evidence and we scale those, what are the potential drawbacks if at some point in time, we have more evidence that suggests this really doesn't work.
Wynne Norton: So kind of having to D implemented based on lack of support or lack of empirical evidence, when we start, and I think the dare project.
Wynne Norton: or dare program drug abuse resistance, education, which I was exposed to in in middle school, I think you know, is an example of going to scale successfully and then having to D implement it so that's kind of the caveat that I would put on that.
Brian Mittman: Great so let me turn to Dr halpern and point out, by the way, that in addition to what medical and PhD and mph degrees and training and that diverse set of perspectives and.
Brian Mittman: Dr Hawkins research background includes you know healthcare delivery clinical settings as well as policy and practice research settings prior nci so when.
Brian Mittman: just about every single stakeholder group and type of institution and setting the weekend, ask for to answer the following question, and that is.
Brian Mittman: You know the question of multi level intervention research that involves implementation activities, well then, when we think about costs and how to segregate.
Brian Mittman: and classify costs and how to measure and then report them both as researchers, but also when we present them to policymakers and practice leaders, you know, the question is.
Brian Mittman: You know how do we.
Brian Mittman: You know, identify intervention costs, you know temporary implementation costs that have to do with the initial training and reorganization and construction than their long term maintenance costs that may differ, both on the intervention, as well as the.
Brian Mittman: Implementation side, so are there some simple readily accessible frameworks that provide guidance in thinking about that.
Michael Halpern: Thank you, Brian and i'd like to thank the human Erica for for inviting me to be here, I.
Michael Halpern: appreciate you being voted to speak to this group.
Michael Halpern: i'd like to point out that, with all my training, I am not formally an economist, I actually don't have a degree in economics, I just play one on the Internet.
Michael Halpern: But much of my research has been in health economics and cancer economics in.
Michael Halpern: A field, I continue to work.
Michael Halpern: And I think that that's a great question and it's clearly very important for implementation science.
Michael Halpern: There isn't a simple answer to it, but I think in terms of a framework.
Michael Halpern: One way to think about it, is to think about the direct costs in any intervention in terms of three buckets study costs startup costs and continuing or ongoing costs.
Michael Halpern: So when you are doing intervention there's going to be study costs costs associated with studying the intervention that because you are doing a research, study.
Michael Halpern: you're going to incur some costs that if it was say a normal part of clinical or public health practice wouldn't occur now clearly you want to be able to separate those costs out.
Michael Halpern: Because those, as I said, wouldn't be part of a normal clinical or public health program and you don't want to mix those costs and you want to be able to very carefully segregate those costs out because they're only being incurred for research purposes.
Michael Halpern: The other two groups kind of the startup costs and the ongoing or continuing costs are little trickier because you want to be able to identify both of them that they're they're both important for any kind of ongoing program but you do want to be able to separate them.
Michael Halpern: Because they are going to have different implications for policymakers and for healthcare delivery purposes the startup costs are those associate as the name suggests, with starting up an intervention for.
Michael Halpern: When you initially implement a program and they tend to be higher, what do you need to do to get buy in to get Community engagement to.
Michael Halpern: activate intervention versus the ongoing costs the costs that would go that when intervention or programs it steady state and so you'll want to be capturing both kinds of costs.
Michael Halpern: When you're working on some sort of intervention, but ideally you'd be able to separate the two different groups so that you can report to.
Michael Halpern: policymakers to other kinds of audiences, where the two groups are so there'll be aware what kind of close they might incur when they start a new program versus when a program is ongoing.
Brian Mittman: Right matt put an important question in the chat, and that is, you know Ideally, we would have continuous program evaluation for any innovative program that is implemented, the plug within.
Brian Mittman: Healthcare public health setting, we know that doesn't always happen, but if that is part of the plan is that considered a program delivery cost, or should it be segregated and you to somewhere between program delivery and research.
Michael Halpern: Oh no absolutely that's a great question and continuous program evaluation should certainly be part of the program costs and I, I strongly agree that that should be part of the program and that should be part of the program cost.
Brian Mittman: Okay, let me shift back when and your direction and ask about the issue of evidence for an innovative Program.
Brian Mittman: As that program evolves over time during the sustainment and then scale up and spread process and during the inevitable, you know adaptation local tailoring and just general drift of a program.
Brian Mittman: You know, are there any rules of thumb is to when you know, a newer version of Program.
Brian Mittman: is different enough from the original that no longer can be considered the same evidence based intervention and how do we deal with that both as researchers, as well as practice leaders who were interested in, you know evidence based practice.
Wynne Norton: yeah it's a great question.
Wynne Norton: And I think it kind of dovetails into this cyclical approach of intervention development implementation of monitoring the adaptation evolution and scale up and spread in sustainability, so I think.
Wynne Norton: Again, it depends.
Researchers favorite response.
Wynne Norton: But you know, I think it is dependent upon.
Wynne Norton: Collecting you know, monitoring and evaluation data that are collected either either as part of a research program you know testing scale up strategies, for example, or support it.
Wynne Norton: As part of ongoing you know continuous quality improvement or whatever, however it's been assessed in what context.
Wynne Norton: If the target outcomes change, I think that might be significantly different if an either warrant you know, identifying an intervention that focuses on different targets that'd be a better fit so.
Brian Mittman: kind of.
Wynne Norton: I guess you know it would technically be a different you know multi level intervention or different you know patient level intervention.
Wynne Norton: I think you know it may be a fit include includes or needs to include different modules or different types of multi level interventions, if you swap those out so long as there's evidence for that.
Wynne Norton: So I don't know again if there's kind of a here's the line it's either you know here's a new intervention and continued the old one, I guess it depends on what your research question is if you're studying that.
Wynne Norton: And why and what you think a new intervention would be warranted.
Wynne Norton: That would be kind of above and beyond just adapting that intervention, so if you're again, including different components, or if you need to do something entirely different you know sunset that intervention and replace it with something new.
Brian Mittman: And of course i'll take the opportunity to plug concepts of core functions and forms is a partial answer this question as well, that if we do what the.
Brian Mittman: conceptualize and describe and deliver on these interventions in terms of their core functions, you know that allows us to smooth out a lot of the local adaptations and tailoring.
Brian Mittman: That are not consequential that if we're just swapping different forms, but we have the same core function operationalize.
Brian Mittman: You know it's completely legitimate to view that is the same intervention, where the evidence from the prior studies still applies.
Brian Mittman: And i'd like to combine two questions, because for each of you in fact it's the same question, and that is when do we begin to think about measuring costs and thinking about economic questions and thinking about implementation barriers and issues when we're developing.
Brian Mittman: A new intervention and the basic argument is during the early phases of intervention development, you know, we still have a lot of rough rough edges, we still have a lot of.
Brian Mittman: aspects that we haven't quite worked out the intervention is likely to change in significant ways, between now and when we get to the point where we've finished our effectiveness evidence and are ready.
Brian Mittman: To you know do a proper full scale economic evaluation and implementation effort, but on the other hand, we don't necessarily want to.
Brian Mittman: Do all of that developmental work and develop evidence for something that we know or should know from the beginning is not implementable or it's just too costly.
Brian Mittman: So, you know how do you again we'll start dumb and Michael with you think about measuring costs during a very early stage of research, where you know the costs are not likely to be that representative what we will ultimately see.
Brian Mittman: y'all in the final version, and yet we don't want to proceed in a vacuum.
Michael Halpern: it's it's a great question.
Michael Halpern: And, and I agree it's important to get some estimates.
Michael Halpern: One of the I helped lead a conference in December 2020 that was hosted by nci called the future of cancer health economics research and one of the topics that was really emphasized at this conference was the need for increased work in modeling.
Michael Halpern: that we need to do more in terms of looking at costs, particularly costs related to cancer interventions.
Michael Halpern: In terms of models that much of the work is being done with interventions in collecting perspective data and that's wonderful, but there is a tremendous opportunity to do models.
Michael Halpern: And I think I think this is a the question really focuses on that that when a intervention is being in early stage development has been roughed out.
Michael Halpern: may change substantially it's a ideal opportunity to estimate costs using models to take data from previous interventions.
Michael Halpern: To estimate how those costs could be applied to the intervention that's under development what the likely cost of our.
Michael Halpern: will likely savings associated with it might be an associate outcomes might be, and just begin to make projections.
Michael Halpern: Doing number of sensitivity analysis scenarios looking at best case and worst cases and just begin to get a sense of the range and whether or not it's going to be feasible and sustainable.
Michael Halpern: And just begin to get kind of those boundaries as to what are going to be the driving factors in terms of the overall cost of the intervention.
Brian Mittman: that's very helpful and I think one point that i'll pass along it's directly relevant that I often hear from.
Brian Mittman: health system leaders and Kaiser, and that is your don't automatically assume that something which is more expensive than what we currently do is not feasible.
Brian Mittman: That you know, we need to look at the benefits as well and health system will be more than willing to incur additional costs if the benefits are sufficient.
Brian Mittman: So you know we shouldn't immediately assume that we shouldn't proceed with something I think the same may be true of implementation issues where, if something is likely to require.
Brian Mittman: significant amounts of training significant amounts of behavior change a yell in a very intensive implementation intervention to hope to get any form of adoption.
Brian Mittman: If it's beneficial enough, you know it would warrant the investment when the other thoughts comments on on the issue of when is it too early to think about implementation and how do we think about it when we have a very early stage version of the intervention.
Wynne Norton: yeah I think it's it's again a great question and I just echo what Michael was saying in terms of projections.
Wynne Norton: and
Wynne Norton: dog is coming back from daycare yes, I send my dog today um.
Wynne Norton: You know, I think it it isn't too early, I think one of the questions you know clinical staff and clinical leadership will say, well, as this reimburse level.
Wynne Norton: And even if it's not currently I think you know you can always have an eye toward that so telehealth as an example, I don't think anyone could have predicted coven and the.
Wynne Norton: Increase in tele health, and you know the potential for it to be sustained partly given you know the infrastructure and some of the challenges with.
Wynne Norton: Rural patients and settings and transportation challenges, for example, so you know, again, I think it it's important to think of it early and often and.
Wynne Norton: At least an implementation science nowadays we're starting to think about you know how efficient and cost effective are some strategies or combinations of strategies relative to others.
Wynne Norton: and
Wynne Norton: You know if you get more bang, for your buck, so to speak with one collection versus another in terms of how effective it is Brian as you were saying.
Wynne Norton: about reaching those implementation outcomes, for example, and how quickly you can do that as well, so maybe a bundle of strategies is more costly and takes longer, but overall it's effective or more effective compared to fewer strategies, for example.
Brian Mittman: So I think one key implication is the worst thing we can do is ignore these issues and not think about them all together.
Brian Mittman: Perhaps second worst thing that we can do is make some assumptions or priori about what is or is not feasible or valuable so having the information without necessarily prejudging what then information needs.
Brian Mittman: allows us to proceed and make an incremental decisions and decide as the evidence begins to emerge that.
Brian Mittman: Yes, this is perhaps a bit more expensive or difficult to implement, but you know the indications are that it's sufficiently effective and valuable that maybe that's worthwhile or.
Brian Mittman: This appears to be more expensive more difficult to implement and it doesn't appear to add that much more value than what we have already, namely cheaper or easier methods so maybe it doesn't warrant further investment and i'll emphasize the may be.
Brian Mittman: Again, important not to prejudge.
Brian Mittman: Other.
Brian Mittman: Questions via chat or anyone who'd like to unmute and pose any questions for our two panelists are two experts.
Brian Mittman: or any important points additional points from the Faculty given your prior experience dealing with these issues.
Michael Halpern: i'd like to bring it up what one point that came about from the conference I mentioned back in December 2020.
Michael Halpern: While the conference was called the future of cancer health economics, research, one of the key takeaways from the conference was that much of the research in the field.
Michael Halpern: is performed by individuals who, although having economics training don't have graduate degrees in economics and that.
Michael Halpern: Health economics research and particular health economics research related to cancer is really a transistor field it's not a discipline or sub discipline of economics.
Michael Halpern: And that people who work in this field have primary training in a range of disciplines and a very diverse area and one of the conclusions of the conference was that, for this to grow.
Michael Halpern: it's absolutely critical that we.
Michael Halpern: Welcome and engage individuals from a range of disciplines and that we have opportunities and training for for people from lots of different areas, so if individuals who are.
Michael Halpern: In implementation science are very good in working this area, I want to encourage you to learn more to be involved in economics research and to not let.
Michael Halpern: The name economics scare you off or dissuade you from from to work in this area.
Brian Mittman: Right very helpful and Dr Hudson has a question.
Matthew F. Hudson: or comment, colleagues, Dr delbert Thank you very much for your comments.
Matthew F. Hudson: picking up on your observation that health economics is a trans disciplinary enterprise that that operates somewhat independent of traditional economics.
Matthew F. Hudson: Would you speak to perhaps a fundamental assumption or set of assumptions that health economists make that a traditional economist may not.
Michael Halpern: um.
Michael Halpern: I that's.
Michael Halpern: interesting question.
Michael Halpern: I think.
Michael Halpern: One of the.
Michael Halpern: Differences is that health economists very much look at the role of structural and policy factors and I don't think that's anything novel or new to this audience but.
Michael Halpern: Health economists and typically those who do anything related to cancer care.
Michael Halpern: are very interested in issues related to.
Michael Halpern: Health care providers healthcare systems overuse under use and misuse of care services.
Michael Halpern: Barriers to use among underserved populations, and I think consideration of these kinds of issues and of policies and regulations that affect these issues.
Michael Halpern: isn't kind of the bread and butter that many economists in in not health issues really focus on but but it's really critical issues within health economics.
Michael Halpern: So I think that's kind of one of the main differentiating factors, for you know why economics is really different and is transistor.
Matthew F. Hudson: Sorry, I just just a follow up question that that it seems to me that a traditional economist may put the focused on a consumer as a patient where a health economist May.
Matthew F. Hudson: be thinking more critically about.
Matthew F. Hudson: The the consumer being a system or an organization is that is that a fair assumption.
Michael Halpern: I think that's certainly part of it, yes and and that.
Michael Halpern: You know it's the you know there's an incredible relationship it's the patient and the provider and the health care system and the you know the healthcare system being a.
Michael Halpern: physician practice and a hospital and medicare or medicaid or a private ensure or the country as a whole, so it's a much more complex system than many other kind of economic issues.
Matthew F. Hudson: Thank you, Sir.
Brian Mittman: And I assume that some of the general principles that we think about when we think about you know any sort of.
Brian Mittman: Translation or adaptation of other sets of ideas and, in my case it's you know ideas for management and y'all from other sectors, the economy, did healthcare that thinking through those would apply, so we know that healthcare as a service, you know, has.
Brian Mittman: Has has a lot more uncertainty, you know we don't have the ability to know exactly what to do and exactly what it will produce health care.
Brian Mittman: should be thought of as a right, rather than a consumer good where you know supply and demand and the option to purchase.
Brian Mittman: Is there, well, we know that we have situations where the consumers aren't paying the full cost, we have insurance.
Brian Mittman: You know the frontline workers are professionals with autonomy, rather than your frontline assembly line workers who follow the organizational.
Brian Mittman: Standard operating procedures and all of those yelp complicate you know the kinds of approaches will be us in analyzing yield management issues and presumably.
Brian Mittman: Health economics as well, so is that a fair characterization that we have to think about some of those fundamental differences and thinking about the adaptation of standard economic approaches to health economics.
Michael Halpern: Thank you, Brian that those are perfect yes it's as opposed to many other kinds of commercial, economic transactions and healthcare transactions are not are not balanced as we've seen repeatedly it's not a there's a power differential and.
Michael Halpern: That certainly affects it.
Brian Mittman: Okay we're just a little bit over and, as usual, we can continue talking about these issues for quite some time but we'd like to split into small groups so.
Brian Mittman: Again, our thanks to a or to distinguish stuff factual the Faculty lecture is not only for your lectures, but also joining us for the panel and Eric any additional thoughts before we split into our groups.
Erica Breslau: I think this was fantastic having you both, and thank you so much for joining us Michael, would you be able to post, the link to the meeting that you had in December, I think.
Erica Breslau: People might find it very interesting and i'm not and i'll also add that Michael has done a journal of the national cancer institute special issue that I think is coming out in July or August Am I correct.
Michael Halpern: In July.
Michael Halpern: It will be published next month, yes yeah.
Erica Breslau: So I encourage you to do a web search on that and we'll send it to you it's a lot of the thanks Michael it's a lot of the.
Erica Breslau: Thanks Brian it's a lot of the proceedings so keep your eye out for that and we are asking both Michael and when to join small groups.
Erica Breslau: So I think we're ready and nita.
Anita Peterson: Okay well do.