Are decisions well supported for the energy transition? A review on modeling approaches for renewable energy policy evaluation

Abstract

This study reviews energy policy evaluation approaches on their capability to estimate a successful implementation of renewable energy policies. This is predominantly done via energy system modeling and analysis. Although modeling the possible success and effects is not a precondition for policy making, it is a powerful tool to support decision makers in policy making. This awareness has led to the development of numerous modeling approaches with many case studies. Therefore, effort has to be made to evaluate recent modeling approaches that could be suitable for renewable energy policy evaluation. It is the aim of this paper to provide an overview on recent renewable energy policy modeling approaches that are capable in evaluating the success and side effects to other sectors of renewable energy policies. We will highlight advantages and drawbacks of these approaches and provide a framework assessing the suitability of the presented methodologies for the evaluation of renewable energy policies. We provide a tabular overview that enables the reader to quickly derive information on the suitability of the several modeling approaches to evaluate renewable energy policies.

Review

In 1980, the term Energiewende was first used in a study of the Öko-Institut [1]. Since then, several policies were installed to follow the idea of a decarbonized energy system. This includes among others the support of the development of renewable energy and associated technologies, efficiency actions, renewable fuels, reduced import independence from fossil fuels, rural development, the decrease of long-term economic costs in kind of external effects, and R&D support [2–4]. International Energy Agency (IEA) defines renewable energy as energy derived from natural sources like sunlight, wind or some forms of biomass in terms of being not finite like fossil fuels having simultaneously a smaller impact on the environment [5]. Interplay between renewable energies, energy efficiency like green buildings, and a sustainable transport sector is crucial for achieving the imposed goals of the German Government and the Energiewende. Renewable energy is characterized by replenishing at a faster rate than being consumed. Like Germany, several countries implemented strategies, policies, and stimulus packages to promote the transition towards a renewable energy system and increase their overall share of renewable energy in the energy system. However, increasing the share of renewable energy is often associated with sustainability challenges [6, 7]. The key elements of a sustainable development like economic performance, environmental protection, and social responsibility need to be considered in designing or evaluating support schemes, policies, or stimulus packages. But the design and evaluation of renewable energy promoting policies is a difficult task considering the multiple aspects policy designers and decision makers have to consider. Governmental interventions aiming at a renewable energy system promotion are expected to fulfill numerous demands but simultaneously affect many areas of human being.

One problem often arising with the implementation of renewable energy promoting instruments is the concentration on one energy sector. For example, in Germany, there are policies promoting the substitution of fossil energy resources in the power sector, like the Renewable Energy Source Act. Biofuel quotas promote the usage of renewable and sustainable fuels. In the heat sector, there is an obligation to use renewable heat for new constructed buildings on state level with an extension to the already existing building stock in one federal state. Each of these policies promotes partially a renewable energy system, the substitution of fossil resources, and an overall low-carbon energy system. However, each of these instruments has a restricted view on its effects, limited to the sector in which they should promote renewable energy. But there are side effects to the other sectors like the promotion of renewable power in Germany that in the recent past has led to an increase of power produced by coal plants having equalizing capacity due to the merit-order effect [8]. Renewable excess power from wind and photovoltaic power plants for example can be stored as gaseous energy carrier (i.e., hydrogen) and used later in combined heat and power plants or as fuel and thus affecting the heat and fuel sector. Hence, there are side effects promoting a renewable energy system.

In recent decades, policy advice has built on the results of mathematical models [9]. The advantages of a combination of computational power and theoretical analyses are only one driver for the widespread use of mathematical models. Furthermore, this follows the idea that decisions are no longer driven by subjective impressions but by profound technical analyses, what Porter called “the pursuit of objectivity” [10]. Although this aim is still a bit up in the air, it has led to efforts of regulators and model developers to analyze and evaluate models. Following MacGillivray and Richards [9], these analyses and evaluations focus on issues including the plausibility of modeling assumptions, precision, and bias; the adequacy of the treatment of uncertainty and the value judgments that models may implicitly or explicitly encode. They sum these factors up with the term model quality. Determining the quality of a model is a difficult task. There are no formal modeling standards you can compare a model with. To assess the quality of a model is therefore a subjective action. A more impartial way is to assess model quality by comprehensible measures like accuracy, clarity, completeness, conciseness, or consistency. It is important that a model has a message and that this message is fully conveyed.

One problem arising when the effects of many different policies are evaluated by many different modeling techniques is an almost confusing and unclear muddle of possibilities to choose from. In the research literature, there are a number of efforts to evaluate or compare models with regard to energy, but a systematic and comparative study is rarely found. First effort was made by Hoffman and Wood [11] focusing on the evolution and development of energy system models followed by, among others, Panday [12] and Urban et al. [13]. Although they have carried out a comparative study, their focus was different from ours. In this study, we want to highlight modeling methodologies that have been used to test policies with regard to the introduction and dissemination of renewable energy. For instance, Teufel et al. [14] provide a review of system dynamics models for electricity market simulations. Within their extensive model review, they identified among others the trend in system dynamics market simulations to simulate more detailed new aspects and markets like CO₂ certificates, green electricity certificates, and integration of RE sources. Besides electricity, green buildings play a major role in achieving the national and international goals of governments to reduce greenhouse gas (GHG) emissions and to increase the energy efficiency. In Coakley et al. [15], a literature review on methods modeling, building energy, and simulating cost-effectiveness of energy conservation measures was done. The industrial sector, being a sector with huge energy-saving potential, was part of a study of Olanrewaju and Jimoh [16]. They reviewed energy system models to develop a tool for long-term planning of energy supply that is indeed a pillar of the energy revolution. One approach to deal with decision-making is decision analysis in terms of multiple criteria analysis. Wang and Poh [17] did a survey on decision analysis in energy and environmental modeling. They show the evolution of this modeling technique and recent developments [17]. Summarizing it can be said that numerous models and modeling techniques were surveyed and reviewed that are connected to fields of modeling renewable energy policy evaluation. While all these studies are doubtless useful, a clear gap in the knowledge becomes obvious. What is non-existent in the current research literature is a systematic overview on policy evaluation modeling techniques determining their suitability for renewable energy. This study aims to close this knowledge gap. Accordingly, the objective of the paper is to provide a systematic overview of a set of modeling methodologies capable of evaluating renewable energy policies. We focus on renewable energy alone because their implementation into the energy system is often accompanied by inter alia overcoming market failures. However, their market penetration can be observed globally and is without alternative.

This paper provides a framework to help decision makers or scientists to decide on one policy evaluation modeling approach depending on the question they want to answer regarding the strengths and weaknesses of the several approaches. It is highly advisable to foster a plurality of different modeling approaches as this article is not about choosing one approach over another. Thus, a systematic overview on recent approaches will be provided and highlight which modeling approach fits to which scope of application. It must be said that there is not the one and only modeling approach being free of restrictions. In this paper, we want to answer the question which policy evaluation approach fits best to analyze future policies in the context of the Energiewende. Due to the fact that different policy results affect each other, it is highly advisable to estimate possible outcomes of future policies through modeling and simulation techniques. We combine an extensive literature and model review to assess the characteristics of the different approaches. It is shown that there is not one approach that is the best one but each approach has its advantages and drawbacks in policy evaluation modeling. Nevertheless, hybrid methods tend to are the most balanced ones. A combination of system dynamics and agent-based modeling is a promising approach.

Methods

Literature selection and categorization

The first step of our research was an intensive literature review to identify the most important policy evaluation approaches and models. The identified approaches were grouped following Spyridaki and Flamos [19] and Möst and Fichtner [18] (Fig. 1) and encompass input-output analyses, computable general equilibrium, system dynamics, multi-agent, multi-criteria analysis, theoretical analysis, and hybrid approaches. We decided to distinguish between quantitative, qualitative, and hybrid approaches. Quantitative methods were further distinguished between top-down and bottom-up methods. To select case studies for our evaluation approach, an analysis of the research literature was done using databases of ScienceDirect and Mendeley. Different keywords in titles and abstracts were used to identify the relevant articles, such as policy evaluation, policy model, model evaluation, green energy model, and renewable energy model. Because of the relatively young history of renewable energy models and with this renewable energy policy evaluation, the starting point for the search is 2005 with the article Harmelink et al. [65]. The final point for our search was 2016, with the article Li and Jiang [30]. In total, 180 articles were identified. Further investigation was made by using the following criteria to sort out inappropriate studies:

Fig. 1

Classification of policy evaluation approaches [18, 19]

• Clear assignment to one of the modeling approaches illustrated in Fig. 1

• Modeling technique and assumptions have to be published in detail

• Traceability of modeling approach

• Publications in peer-reviewed journals

• Recent case studies regarding renewable energy policies

Applying these criteria, 48 case studies remained were used for the evaluation approach. The second step was to identify criteria that can be used to compare the various modeling approaches and to have a basis for the construction of a framework. Applying these criteria, it is possible to deeper investigate the chosen articles, respectively the renewable energy policy evaluation models. Within the analysis of the abovementioned articles, several patterns were observed. As mentioned before, the identified renewable energy (system) modeling techniques were grouped (Fig. 1). To compare the models and their suitability to model and evaluate renewable energy policies, we choose the parameters modeling approach, spatial coverage, sectoral coverage, time horizon, ex-post vs. ex-ante, quality of data sources, assumptions about actor behavior, assumptions about markets and systems, and the possibility to implement the approach in a computer-aided framework. The methods as well as the criteria were chosen with respect to the task of modeling renewable energy policies. They have to meet partially different requirements than modeling techniques used for fossil-based energy systems. This is based on the fact that the introduction of renewable energy is often associated with market failures, subsidies, governmental influence, learning curves, technological progress, cost degression, and etc. It is important that the respective modeling methodology is capable of implementing these factors. More precisely, the criteria for the comparison of the different renewable energy policy evaluation approaches were chosen considering the specific demands of policies aiming at the development of renewable energy. The spatial coverage is an important criterion because policies can be implemented from a regional scale (obligations or subsidies in federal states of a country) up to a global scale (Paris agreement). The sectoral coverage has to be considered because the implementation of renewable energy policies affects many different sectors. In this way, a certain level of disaggregation is important to best possible estimate the effects of renewable energy policies to the different sectors. Time horizon was chosen as criterion because policies can be designed for short term to long term. Besides, these policies can have different effects in the short term than in the long term and vice versa. The differentiation between ex-post and ex-ante is important because it separates between an evaluation of a single policy and a set of policies before (ex-ante) or after (ex-post) the implementation of it (them). The quality of data sources has to be addressed because of the need of a referral system to calibrate the model. The fact how the model reflects the reality is given by the criteria assumptions on actor behavior and assumptions on markets and systems. Finally, the question if the approach could be implemented in a computer-aided framework is considered because this could increase traceability, evaluation time, or the quality of results, just to name a few examples. Questions that shall be asked by a modeler before he decides for a modeling approach to estimate the effects of renewable energy policies comprise, inter alia: Shall the focus be laid on actors (agent-based) or the overall system (system dynamics)? Is there technological progress (system dynamics, CGE, hybrid approach) or not (I/O modeling)? Shall the effects of policies be considered ex-post (qualitative) or ex-ante (quantitative)?

Introduction to energy models for policy evaluation

The implementation of the abovementioned criteria into policy impact analysis frameworks is predominantly done via energy models in a quantitative matter. Another kind of analysis of policy impacts is done via theoretical analysis in a qualitative matter. Furthermore, hybrid forms have been developed. A model is understood as an abstract representation of the real world respectively of a real system. Behavior and interactions of the system elements are described qualitative and quantitative. In this way, a model is a purposeful and simplified reflection of the reality. The main motivation formulating a scientific model is reduction in complexity. Due to extent and diversity of questions and framework conditions, it is obvious that there cannot be one single modeling approach. Just as little, there is one energy system, model there is the “best” methodological approach. The methodological approach is determined by problem formulation and implicit assumptions that were made. To answer the different questions in energy system analysis, various models were used. Based on the diverse methodological approaches and their hybrid forms, a characterization can only be generalized. Long-term energy models are predominantly used for the analysis of energy and environmental policies and politics. Strengths of energy system models are the high level of detail, their flexibility, transferability to new questions, and transparency in modeling results [18]. Energy system models can be separated between optimization models and simulation models. While optimization models try to determine the most cost-efficient option of action, simulation models try to simulate the effects of the given action options.

Although modeling the possible impacts is not a precondition for policy making, it is a powerful tool to support decision makers in policy making. Following Perman (2011), a model is more useful when it is set up inter-sectoral, distinguishing different options of production and consumption [20]. That is done via several types of models namely input-output models, computable general equilibrium models, and linear and non-linear programming (optimization) models [20]. For the difficult task of analyzing the role of energy and climate policies, respectively a set of policies for the development of renewable energies, numerous authors presented their frameworks. The most frequently used models have in common that they assume hyper-rational and homogenous human behavior that ensures tractability but does not consider less-than-rational decision-making. Hyper-rational in the before-mentioned context means that if there was a decision to be made, it is by definition the best choice or else they would have chosen something else. After decades of Integrated Assessment Models (IAMs) that were doubtless helpful to gain insights to mitigation options and climate change dynamics, there truly exist further improvements [21]. The most common modeling approaches for defining and validating a policy or a set of policies are consecutively presented.

We grouped the most commonly used methodologies of policy evaluation design approaches into three categories, namely qualitative, quantitative, and hybrid. Hybrid policy evaluation designs combine quantitative and qualitative attributes. Thereby, the term hybrid is used for a combination of two or more single approaches but not dedicated to one specific combination.

Quantitative designs are handling numerical data within a calculative framework, whereas qualitative ones consist mainly of explanatory descriptions [19]. The major energy and climate policy modeling is done via quantitative methods, namely bottom-up and top-down models. Bottom-up and top-down models are synonyms for aggregate and disaggregate models [22]. Top-down modeling approaches focus on the interaction between the energy sector and the whole economy, whereas bottom-up modeling approaches focus on the energy sector alone. Top-down models, usually input-output models or computable general equilibrium models (CGE), have a closer look at the economy and include inter-sectorial feedback effects. On the contrary, bottom-up models describe technologies in detail, recent, and prospective and come usually as mathematical programming. That is why, they suit best for policies referring to technology changes, like efficiency standards. Drawbacks of bottom-up models lie in their basics. The major drawback, besides their inability to model economy-wide interactions, comes from the mathematical programming itself, i.e., the implementation of tax distortions or market failures [22–24]. Besides this, there are approaches combining bottom-up and top-down. Significant advances in hybrid modeling, meaning the combination of bottom-up and top-down, are made due to the restrictions of both bottom-up and top-down models. The design of hybrid modeling approaches is complicated because of the need of macroeconomic and engineering expertise and data as well as the need of numerical calibration [25].

Input-output models

Input-output modeling as part of empirical economic research bases mainly on the transaction table and was formulated first by Wassily Leontief in 1936. Although input-output (I/O) models were first designed as economic analysis tools, they often contain an ecological component. In the 1960s, first steps were made in linking economy and environment bringing out lots of environmental I/O models in the following years. Policymakers benefit from insights into changing structures of economical basic data like production, consumption, employment, and import/export. If I/O models contain an ecological component, they can also provide information on environmental impact. I/O models can estimate changes to the whole economy made by single changes in economic activities. They are strong in linking production and consumption [23]. Miller and Blair [87] provide a comprehensive guide for environmental and energy I/O models. The integration of environmental aspects into economy-dominated I/O models can be done via coupling with life cycle analysis. Case studies representing the I/O methodology are Bruckner et al. (2005), Pettersson et al. (2012), Liu et al. (2009), Cellura et al. (2013), and Li and Jiang (2016) [26–30].

CGE models

Computable general equilibrium models can show how an economy reacts to changes in policy or technology. They have a neo-classical spirit and are essentially empirical versions of the Walrasian general equilibrium system [20]. CGE models can only be solved computationally, not algebraically. With the increase in computing power and their availability, CGE modeling received a rapid growth. CGE models run mostly midterm to long term. Another approach is presented by Proença and St. Aubyn [31] with their hybrid bottom-up/top-down, multi-sector, static CGE model. It was developed for a small economy and is an extension of Böhringers and Rutherfords model [32]. Case studies of Wang et al. (2009), Wianwiwat and Asafu-Adjaye (2013), Bretschger et al. (2011), Beckman et al. (2011), Kretschmer and Peterson (2010), Fortes et al. (2014), Guo et al. (2014), and Suttles et al. (2014) were used to evaluate the CGE modeling methodology [33–40].

System dynamics

System dynamics models try to reproduce interdependencies between components of a real system considering the time. The mathematical description of the interdependencies bases on differential equations. System dynamics models are predominantly used analyzing liberalized markets. They can display market mechanisms through differentiated mechanisms of action instead of following a single objective function that allows those models a differentiated image of real markets. Equal to other market simulation models, there is the challenge how to suggest plausible assumptions for future behavior of market actors and how this should be transferred into parameters.

With system dynamics, it is possible to analyze interactions of economic and environmental interactions and feedbacks [41]. Furthermore, coupling system dynamics with evaluation tools and methods, it is possible to simulate medium-term effects of potential futures [42]. As Aslani et al. (2014) stated, only little research has been done using the system dynamics methodology for dynamical modeling of renewable energy policies containing energy and climate policies [43]. The reasons for that are the complexities when it comes to combining energy and climate policies and modeling their feedbacks as well as evaluation of results of such a modeling approach. System dynamics with its possibility to analyze the impact of a variable change to the structure of the system can help to evaluate the behavioral change of a policy instrument. That is important because most policy instruments are often evaluated of their impact on development and adoption of innovations instead of their impact on the system behaviors [44]. There are diverse possibilities for analyzing the impact of energy and climate policy instruments within a system dynamics framework. After achieving confidence in the system dynamics model, i.e., through evaluation with historical data, the analysis is mostly done via experimentation, exhaustive what-if-scenarios, and automatic optimization via external software [45–47]. The main activities within these kinds of analyses are trial-and-error simulation, changing parameter values or switching parameters, and loops on and off [44]. Because of the inherent limitations of these methods especially for large models with 100+ variables, Al-Saleh and Mahroum (2015) developed a method for structured policy analysis. This method combines linearization of the model, the dynamic decompensation weights analysis (DDWA) and the loop eigenvalue elasticity analysis (LEEA). This composition of methods provides a circular process of testing, policy analysis, and policy interpretation for implementation [44]. System dynamics methodology is used by Hsu [48], Barisa et al. [49], Li et al. [50], Aslani et al. [43], Jeon and Shin [51], Wang et al. [52], Wu et al. [53], Chyong Chi et al. [54], Szarka et al. [42], Movilla et al. [55], and Ansari and Seifi [56] to model renewable energy policy evaluation. These references were used to evaluate the methodology [42, 43, 48–56].

Agent-based modeling

The method of agent-based modeling enables a researcher to analyze individual behavior in a system. Actions can be simulated on an individual scale, and unique decisions can be observed that cannot be identified on a system scale. So-called market participants are represented by agents. Each agent has kinds of assets that can be made as detailed as in energy systems. In markets, each agent follows his own bidding strategy defined by his individual objective function. Agents communicate with each other and adapt their strategy on market developments. In contrast, multi-agent models focus on solving specific problems using independent agents, while agents in agent-based models focus on whether agents are obeying assigned rules that forms their behavior. Gerst et al. [21] developed the agent-based model ENGAGE including a scenario discovery tool and a policy discovery tool. A detailed overview on agent-based simulation models is presented in [36] and [57]. Agent-based approaches for energy system simulations are usually used for short-term meditations in liberalized energy markets. Lee et al. [58], Nannen et al. [59], Tang et al. [60], Gerst et al. [21], Zhang et al. [61], Gerst et al. [62], and Ding et al [63] are case studies using agent-based modeling methodology. They were used for the evaluation of the methodology [21, 58–63].

Theory-based evaluation

An extensive description and illustration of the methodology of theory-based evaluation is presented by Rossi et al [64]. At first, the method was not used for energy policy purposes [64]. It is now used for ex-post policy evaluation unraveling the whole policy making and implementation process to determine what went wrong and which improvements to more efficiency and effectiveness can be made. Theory-based evaluation (TBE) approaches are ex-post evaluation approaches providing insight on success and failure of policy instruments [65]. They pay attention to the theories of policymakers and other decision makers. They take a deeper look at hypotheses that are empirically testable. These theories base on intervention logic, meaning that policy actions intend a certain outcome, i.e., climate protection or energy security when using financial resources as input. The effectiveness of a policy can be measured by outcomes. A theory-based evaluation consists of mainly two compounds. The first is conceptual, articulating a policy theory. The second is empirical, prospecting to test the theory and to find out how and why the observed outcomes come off. The theories that underlie a policy program are often not directly visible or even documented. Policy evaluators have to find and test the theories. This can be done with varying approaches within theory-based evaluation:

• Realist evaluation focuses on the importance of CMO (context, mechanism, outcome)

• Theory of change considers the importance of short and medium-term effects on long-term effects as well as the connections of programs and outcomes at each step

• Contribution analysis tries to identify whether an outcome was produced by a certain program or external effects

The approaches of Harmelink et al. [65], Harmelink et al. [66], Murphy et al. [67], and Abdul-Manan et al. [68] are case studies representing the theory-based methodology. They were used to evaluate the methodology [65–68].

Multi-criteria analysis

Multi-criteria analysis (MCA) is part of decision theory and a collective term for any structured approach determining preferences under given options. The options get ranked differently on the criteria that are considered relevant. MCA is a mostly subjective approach founding on the decision makers’ evaluations, although objective data like prices can be added. There are many diverse MCA methods from the simplest method weighted sums to more complex or even hybrid approaches. MCA is used when a single criterion decision analysis approach like a cost-benefit analysis is insufficient. MCA can include social, environmental, technical, economic, and financial criteria. The output of a MCA is a single most preferred option respective a ranking of options. The case studies of Browne et al. (2010), von Stechow et al. (2011), Clo et al. (2013), Yavuz et al. (2015), and Cannemi et al. (2014) represent the theory-based methodology with regard to the evaluation of renewable energy policies [69–73].

Hybrid approaches

As mentioned before, the combination of at least two modeling approaches is called hybrid. The term is mostly used for a combination of bottom-up and top-down models. Combined bottom-up and top-down approaches can be categorized following Böhringer and Rutherford [32]:

• Coupling existing large-scale bottom-up and top-down models [46]; Böhringer and Rutherford call this a “soft-link” because of the substantial problems in consistency and convergence of iterative solution algorithms. Therefore, this approach may face substantial problems

• The second approach deals with the combination of one complete model type with a reduced form of the other one. Commonly, bottom-up energy system models are linked with an aggregated one sector top-down model [74, 75].

• More recent is the third approach that combines bottom-up and top-down through the characteristics of market equilibrium models, called mixed complementary problems (MCP) [32, 76]

While most quantitative methods can perform energy and climate policy evaluation on their own, qualitative approaches are often combined. According to Boonekamp (2006) and Sorrell et. al (2003b), there is a need to combine quantitative and qualitative methods for complex tasks [77, 78]. Boonekamp [77] used a qualitative analysis with a bottom-up energy system model to evaluate interaction effects for policy instruments for household energy efficiency in the Netherlands for 1990–2003. Further hybrid approaches combining quantitative and qualitative evaluation methods were done by [32, 79]. Sarica and Tyner [80], Igos et al. [81], and Cai et al. (2015), Proença and St. Aubyn [31], Strachan and Kannan [83], Jaccard et al. [84], Böhringer and Rutherford (2008), Pollitt et al. [85], and Barker et al. [86] are case studies that used a hybrid modeling approach to evaluate renewable energy policies. They were used to evaluate the methodology of hybrid modeling approaches [31, 32, 80–86].

Input-output model evaluation for renewable energy policy modeling

To evaluate the I/O methodology to its applicability to renewable energy policy evaluation, several case studies were investigated. Table 1 summarizes the evaluation results. In case of renewable energy policy analysis, I/O models can be a helpful tool in problem analysis, monitoring, and ex-ante policy analysis. Within all investigated case studies, it is obvious that the I/O modeling approach has its strengths in economy-related issues. Most case studies included explicit and extensive price calculations as well as various fuel stocks. Technological progress plays no role in I/O modeling as well as dynamic markets. Recently, it is possible to integrate I/O analysis in computing frameworks like the statistical computing software R. I/O analysis allows the comparison of scenarios like the implementation of a carbon tax vs. the implementation of an emission trading scheme. Thus, this makes it suitable for environmental policy analysis. I/O analysis is usually used for short-term estimation of policy implementation effects. Long-term forecasting and simulations cannot be derived from I/O analysis. Having their advantages in being transparent and computationally straightforward, the main drawbacks of I/O models lie in the focus on short-term effects and the non-capability of feedback effects [87]. Furthermore, utility- and profit-maximizing behavior as well as dynamics play no role [20]. To overcome at least the drawback of nonobservance of dynamics, Masouman [88] suggests coupling I/O models with econometric models. The reliability on data can be assumed as medium. It is possible to have an unbalanced data set with missing observation points to use trend estimation for prices where actual data is missing and assume that energy and capital inputs are comparable to labor and material inputs, because of missing data for labor and material [27]. I/O models can be used for regional as well as global analysis.

Table 1 Evaluation results of methodologies for renewable energy policy modeling

CGE models evaluation for renewable energy policy modeling

To determine the applicability of the CGE methodology for evaluation modeling of renewable energy policies, several case studies were investigated in detail. Table 1 shows the evaluation results in more detail. CGE models are mostly used for economic research questions. When investigating the results of environmental policies, CGE models were coupled with environmental indicators. This is mostly done for air pollutants like CO₂ [40]. Previously, the combination of equations and databases describing a CGE model were often solved in a custom-written program. Today, most CGE models are embedded in known software systems like GAMS, Matlab, or Excel. This comes along with a cost-reduction for CGE modeling as well as an increase in transparency. Within the CGE approach, it is possible to derive information on success and failure of policy adjustments for short-term to long-term simulations [34]. Because one CGE model is usually set up for one country or region, transferability of one approach to similar ones is tough. Drawbacks of most CGE models are a low degree of detailedness and a higher but still insufficient attention of dynamics. In addition, CGE models assume full price adjustment and equilibrium in all markets, including the labor market, with the effect that there will be no compulsory unemployment, resulting in the fact that results tend to be determined by the supply side rather than the demand side [85]. Mostly, they lack in providing technological details and development. Furthermore, they have problems in assuming how technologies will evolve in the future as well as future cost-development. Additionally, most top-down models violate fundamental physical restrictions such as the conservation of matter and energy [22]. Effects of policies on employment are therefore hardly simulated. Being post-Keynesian in nature, the E3MG model, numerously used for the simulation of effects of renewable energy policies, is able to overcome the before-mentioned shortcoming by being based on the theory of effective demand. It has to be mentioned that using dynamic econometrically estimated equations sets the E3MG model apart from usual CGE model that are static [34, 40]. Following Barker [86], the E3MG model as well as the E3ME are considered as a hybrid model. An argument is the underlid behavior of consumers and producers which is often not met [34]. Furthermore, the needed data is frequently unavailable and has to be assumed, what is another drawback of CGE models [39]. Being often criticized for not having a sufficient validation, CGE models can often be seen critical. A novel approach with a calibration was done by Beckman et al. [34].

System dynamics evaluation for renewable energy policy modeling

To evaluate the system dynamics methodology to its applicability to renewable energy policy modeling, several case studies were used. The results of the evaluation are given in Table 1. The methodology distinguishes itself from a high level of traceability. The system dynamics methodology is characterized by being a modeling technique fitting to a broad field of research questions [56]. With a combination of optimization approaches like what Al-Saleh and Mahroum (2015) presented, a comprehensive policy mix optimization could be added. In this way, the most sustainable and successful mix of policies could be derived.

Qualitative system dynamics models like causal loop diagrams reflecting cause-effect relationships can be set up without a simulation or computing environment (Ansari and Seifi, [56]). The transfer of a qualitative to a quantitative system dynamics model is usually done in software environments like Stella, Vensim, or Powersim because of a better visualization ability and faster solving of the implemented differential equations [56]. Within the system dynamics approach, midterm to long-term simulations can be computed comparing different scenarios [50, 55]. System dynamics modeling distinguishes oneself from a high grade of traceability and transferability. Drawbacks of this modeling approach lie in validation of the interdependencies, defined by the model developer and the necessity of calibrating the model with suitable data, i.e., historic data [55]. If there is no real referral system, a calibration with real or historic data is not possible. Furthermore, system dynamics models are hardly used for modeling short-term effects, because their strength lies in the identification of dynamics in the system that often show their behavior in the mid to long term [48]. A prerequisite for system dynamics modeling is a dynamic behavior of the examined system and interaction between system components through feedbacks, which distinguish it from methodologies assuming a static market, like CGE or partially agent-based modeling [49, 56]. Still, a relationship between system dynamics modeling and agent-based modeling is important to recognize [52]. In addition, system dynamics is strong in considering learning effects and technology innovation [51].

Agent-based models evaluation for renewable energy policy modeling

The research work of several authors was investigated in more detail to determine the applicability of the agent-based methodology for renewable energy policy evaluation modeling. The results of the evaluation are shown in Table 1. Agent-based modeling has a long tradition in social sciences [59]. Typically, agent-based models are set up within a modeling simulation environment and can be used to derive short-term to midterm simulation results [62, 63]. The approach allows the comparison of different scenarios and an effect estimation of the implementation of renewable economy policies [59, 60, 62]. Agent-based modeling is a traceable technique with a low degree of transferability because of the individual programming that is necessary. But if the whole system has to be analyzed and the individual scale is too small, agent-based modeling is not the right choice [58, 62]. The restriction of an identical objective function like it is part in energy system models is abolished by agent-based models [58]. Agents are supposed to do cooperation and an open exchange of information or competition and secrecy [60]. Agent-based models often base on the economic theory of general equilibrium, which is a controversial theory. In addition, agent-based modeling works well for evolutionary approaches [89].

Theory-based evaluation for renewable energy policy modeling

Several case studies were investigated to evaluate the methodology of theory-based policy evaluation, illustrated in Table 1. One problem in theory-based evaluation is data availability. Although other evaluation approaches need a certain amount of data too, theory-based evaluation is highly dependent on data availability [68]. Theory-based approaches (TBA) are not set up in a computing environment. It is not possible to derive simulation-driven results from a theory-based analysis of policy impacts [66]. Rather, this approach deals with the question why a program or policy had or had not an impact [66]. Nevertheless, TBA shows good traceability, whereas transferability is not given due to customized analysis. The strengths of theory-based evaluation are the usefulness in niches were other approaches might fail as well as the property that they have a look at the cause-effect elements and that they can contribute to extent existing data [67, 68]. In the meanwhile, the major drawbacks are that the size of a contribution of an intervention cannot be measured. Furthermore, a theory-based evaluation can be very time- and resource-consuming and needs a lot of data which can be difficult.

Multi-criteria analysis evaluation for renewable energy policy modeling

We found several studies using the multi-criteria analysis for the evaluation of renewable energy policies. Those case studies were investigated deeper to evaluate their applicability to renewable energy policy evaluation. Table 1 shows the evaluation results. The evaluation process within our research shows similar results to the evaluation of the theory-based evaluation methodology. Similarities within the modeling approaches, both approaches are based on decision theory, are the reason for this. But in contrast to the theory-based approach, MCA can be implemented into software environments [70]. This is always a plus for better visualization ability. Scenarios can be compared within the methodology deriving information for short-term to long-term impacts of different policy options [70–72]. MCA approaches are in most cases highly traceable [73]. Due to customized analysis framework, a good transferability is not given. A shortcoming of the MCA method is that it cannot show that doing an action for welfare is better than doing nothing.

Hybrid modeling approaches evaluation for renewable energy policy modeling

Being a novel approach, several recent case studies using different hybrid modeling approaches were used for the evaluation of renewable energy policies. Because qualitative ones perform best for well-specified policy combinations and quantitative ones explain contextual implications and cause-impact effects [19], a combination of approaches from these categories to a hybrid approach is an advantageous concept. Evaluation results are shown in Table 1. Hybrid approaches are predominantly used to reduce drawbacks of a single modeling technique. They are highly able to model economy-wide effects of green energy policies because mainly an improved CGE approach is used as foundation [31, 81, 82]. In addition, environmental-wide effects shall be measured by using an approach that can derive changes to GHG emissions like life cycle assessment (LCA) [81]. Hybrid approaches are a combination of several policy evaluation methodologies, and their applicability in computer-embedded frameworks depends on the single methodologies used [31, 81]. Same is for the property of simulation ability and possible impacts at different time horizons that can be investigated. Furthermore, the traceability can only be judged at every single approach. The complex combination of at least two methodologies being created for a very special problem makes transferability difficult [80]. Whereas bottom-up and top-down methods have their restrictions as presented, the combination of them is not a restriction-free solution. Following Wing [25], it is often a problem that the macroeconomic and the engineering data are rarely consistent with each other. Sarica and Tyner [80] identified three approaches of combining top-down and bottom-up models to overcome the restrictions. These are the following:

• Link independently developed top-down and bottom-up models

• Incorporation of technological detail into a top-down framework or macroeconomic feedback into a bottom-up framework

• Creating a fully integrated model using specific solution algorithms

Another possibility to combine quantitative and qualitative approaches is seen by Söderholm et al. [90] stating that qualitative approaches can be used complementary while constructing scenarios that later can be used for modeling by quantitative approaches.

Summary

As mentioned before, quantitative approaches are more often used for the evaluation of renewable energy policies. Reasons for that could be the characteristic of being able to judge on effects of policies ex-post only, a lower degree of traceability and drawbacks with the implementation in computing environments. An important point for the implementation of renewable energy is technological progress. In this way, it is possible to estimate future market shares. I/O modeling is not capable of including endogenous technological progress, whereas the other ex-ante modeling methodologies are able to do so. Furthermore, long-term forecasting and simulations cannot be derived from I/O analysis. Dynamics is also an important issue but also a kind of attitude of the modeler. Whereas I/O and CGE of the quantitative approaches overlook dynamics, system dynamics is built on the fundamentals of dynamics in a system. Agent-based modeling can pay attention to dynamics if it is set-up as a numeric simulation of multi-agent systems.

Each approach can be used for the simulation of effects of regional policies. Due to its strengths, agent-based modeling is the preferred option to model the relation between agents in markets on a regional scale. Usually, I/O modeling and system dynamics approaches are characterized by a medium level of disaggregation, whereas the other approaches are in need of a high level of disaggregation. I/O modeling is used for the simulation of short-term effects. In contrast, system dynamics shall not be used for short-term modeling because its strengths lie in the simulation of the systems behavior to changes that will most likely occur in the midterm to long term. Agent-based modeling is used for short term to midterm because the behavior of the agents cannot be estimated for the long-term. CGE and hybrid methods can be used for short-term to long-term time horizons. The needed quality of data sources varies between the different approaches. Whereas approaches like system dynamics are mainly dependent on data quality because of the need of a referral system to calibrate the model, I/O modeling and CGE are able to compensate a lack of data quality.

Conclusions

This paper has examined seven common modeling approaches that are applicable for renewable energy policy evaluation, namely I/O modeling, computable general equilibrium modeling, system dynamics modeling, agent-based modeling, theory-based evaluation, multi-criteria analysis, and hybrid approaches. As presented above, quantitative, qualitative, and hybrid modeling approaches are widely used in renewable energy policy planning and evaluation, generating lots of dedicated models as well as model extensions. This is important because in most countries, renewable energy needs governmental support in terms of policies to compete in markets. Nevertheless, there are evident features of each methodology that makes it unsuitable for a specific problem. I/O modeling is favorable for economy-related policy evaluation but has its drawbacks in not considering technological progress and dynamic feedbacks. Like I/O modeling, CGE modeling has its strengths in economy-related policy evaluation issues but struggles with a usually low level of detailedness and dynamics. System dynamics is a modeling technique fitting to a broad variety of research questions with a high degree of traceability and transferability. However, validation of the interdependencies, defined by the model developer, and the necessity of calibrating the model with suitable data are difficulties within the system dynamics modeling approach. Agent-based modeling is a modeling technique which allows the consideration of smaller entities that can behave in a predetermined manner. In contrast, agent-based modeling is not the best choice if the overall system has to be analyzed. Theory-based evaluation can be useful in niches where other modeling approaches fail while having a closer look at cause-effect elements. One major drawback is that the size of a policy implementation effect cannot be measured. Like theory-based evaluation, multi-criteria analysis evaluation is an ex-post evaluation method, strong in niches where the other evaluation approaches might fail. In contrast to theory-based evaluation, multi-criteria analysis can be implemented in computing frameworks. However, its major drawback is that it is not possible to derive information on whether doing an action is better than doing nothing. Hybrid models are combinations of the several different policy evaluation methodologies presented in this manuscript. In this way, hybrid models can minimize (or even abolish) drawbacks of using only one single approach. However, they are not a restriction-free solution.

All modeling approaches have their strengths and weakness, which is why, no modeling approach is superior per se. Policymakers should therefore strive for and promote a certain model plurality. The potential of this plurality is twofold: on the one hand side, decision maker can choose those models which are able to address their specific question most adequate. For the decision of a modeling approach, the relevant criteria are the scale of the renewable policy (local, regional, or national); the time horizon, which should be included (short, medium, or long term); and the granularity of the expected results (technical details, dimensions of assessment, and etc.). On the other hand, the parallel application of different approaches can also be taken to understand the robustness of findings in the different models. Additionally, efforts are undertaken to link the different approaches. One promising avenue for further research is hybrid models that use different modeling approaches in a homogenous context achieving more robust results. Another promising avenue is the development of linkages between already existing models. To sum up, there is not one and only modeling methodology that fits to all requirements for the evaluation of renewable energy policies. The methodologies are as diverse as renewable energy policies due to the broad variety of questions that shall be answered with policy evaluation modeling; thus, a plurality of modeling approaches is highly recommended. However, we presented an overview of the most recent and multiple times used methodologies with information on advantages and drawbacks of each modeling methodology. In this way, it is easier to get information which modeling approach shall be used for which kind of renewable energy policy evaluation model.