Table 2 Full version of the ESS transparency checklist

 1. Author, Institution

−Who are the ESS authors and for which institution(s) do they work?

−John Doe (Scenarios Inc.).

−[40], pp. 2–3 displays information on the study partners, the name of the project manager and lead author, the names and institutes of the research team (co-authors) and the editors.

−In case of a collaborative ESS, what are the authors’ contributions in detail?

−‘John Doe developed the model, while Jane Doe was responsible for the interpretation of the results.’

 2. Aim and funding

−Who are the costumers requesting the study?

−Is there external funding? If yes, what is the source and to which proposal does it refer?

−What is the purpose (hypothesis) and research question of the study?

−What is different in comparison to the current state of research?

−Study commissioned and fully funded by Energy Inc.

−‘Giving policy advice to show whether Nuclear Phase-out in Germany by the end of 2022 is affordable.’

−‘A novel methodology is used, beside new cost parameters which were included.’

−[24], pp. ii expresses that the study is funded by an Agency of United States government.

−[60], pp. I present the core objective of the study which is to show a pathway of moving toward a nuclear-free European Union.

 3. Key term definitions

−How are key terms used in the study’s context?

−Different terms to address the same matter: ‘Grid balancing’ and ‘power system flexibility’ both mean the availability of an energy system to shift power to match supply and demand.

−The same term used for different matters: ‘Demand’ can be interpreted as useful demand or final energy consumption.

−In the context of car propulsion, [61], pp.76–78 clarifies the different automotive technologies. The study also includes a list of acronyms (p.145), but not a glossary.

−Where are key terms defined?

−‘Key terms are defined in the main text’/‘The study includes a glossary section.’

 4. Sources

−What are the sources of empirical data?

−‘Power-plant capacity and spatial distribution base on Enipedia [62].’

−‘Economic statistics, such as electricity or gas prices, are provided by Eurostat [63].’

−[64] Allows online visualization of the data used in the study and the possibility to download the data in an Excel file. For transport, such a file contains information on the units but not the source of data for historical values.

 5. Pre-Processing

−Are empirical data used directly as model input or are data pre-processed?

−How is the characteristic of the empirical data adapted to the input requirements of the model(s)?

−‘The spatial resolution of the empirical data is national energy demand of a country (e.g. Germany). The model needs a spatial resolution for administrative regions (e.g. Berlin). Therefore, the national energy demand is multiplied by distribution factors for each administrative region.’

−[65], p.32 use output from different models and studies and modify with regard to their application. In this specific example, they aim to calculate the total earnings from construction and operation in a 100 % renewable energy based scenario. They therefore use raw, unscaled earning values from JEDI models (Economic Development Impact) and scale them in accordance with their prices to account for changes in wages and labor hours.

 6. Identification of uncertain factors

−What are the (main) uncertain factors and corresponding assumptions (quantitative and qualitative)?

−‘All considered uncertainties concerning future developments can be found in the supplementary section.’

−[66], p.6 describes how the selection of uncertain factors was done, who was included, and which method was use. Pp. 7, 8, 10 list those factors which are explicitly considered in the study.

−Which uncertain factors are explicitly considered in the study?

−‘For all types of power plants a general standard efficiency is assumed according to the following table…’

−‘We take into account lifestyle changes (qualitative factor), as well as the growth rate of the oil price (quantitative factor).’

 7. Uncertainty consideration

−How do you deal with the uncertain factor you have identified?

−‘Lifestyle changes are considered indirectly by their impacts on other factors (e.g. demand and efficiency).’

−‘Oil price is considered directly as an assumption of the model application by an expert’s opinion.’

−[67], pp. 387–389 explain the key driving forces of the scenario study and their assumed alternative developments in the future. They also mention how the assessment was done.

−Which (alternative) development paths are assumed for the uncertain factors?

−Qualitative assessment: ‘Lifestyles could be oriented towards sustainability, materialism or economics by 2050.’

−Quantitative assessment: ‘Oil price growth can be at least 0 %, but not more than 80 % by 2050.’

−‘GDP growth is fixed to 1.6 % per year.’

−What are the sources of the chosen development paths?

−Heuristic.

−Empirical (expert knowledge e.g. 10 economists, 10 consumers, literature review).

−Why and how are individual values chosen from a bandwidth of possible values?

−‘The chosen values are based on own assumptions/desk research/expert workshop/literature review.’

 8. Storyline construction

−What narrative description of scenarios is used which highlights the main scenario characteristics and dynamics?

−What is the relationship between the key driving forces?

−Qualitative narratives can vary in length from brief titles to very long and detailed descriptions.

−[68], pp. 763–764 mention which groups of actors were included in the construction process of the narratives (stakeholder visions) and furthermore describe the methodology, how the visions were created. On p. 766 a description of the applied visions can be found.

−Which method is used to construct the storyline(s)?

−Interdisciplinary expert workshop (social scientist, psychologist, engineer, economist).

−Intuitive scenario construction (no specific method).

−Which normative assumptions are included in the storylines?

−Political targets for green house reductions

 9. Assumptions for data modification

−Which assumptions are related to the modification of model input?

−For the mentioned example in the section ‘Empirical data’: The pre-processing is done by splitting up the national energy demand by population density. The assumption related to this is that the energy demand is proportionally distributed to the population density.

−In [69] pp.30–31 a description is given how the nationally installed capacities of renewable power generation are distributed on a regional level.

 10. Model fact sheet

−Which kind of modeling approach is used?

−Bottom-Up, Top-Down, Hybrid.

−Optimization, Simulation.

−[41], p.85 lists the geographical scope (European Union) as well as the models’ time horizon (1990–2050).

−What is the geographical coverage of the model?

−What temporal resolution and time horizon uses the model?

−Global, 2005–2050, hourly time steps.

−For a comprehensive overview on model classification see also Fig. 2.

−Which sector is addressed by the model?

−Energy (power, heat,…), economy, environment.

−How does the model deal with uncertainties?

−‘The model is deterministic and does not include random elements.’

−‘The model is mainly deterministic but incorporates stochastic components (e.g. includes Monte Carlo simulations based on probability distribution assumptions for key parameters).’

−‘The model is mainly stochastic as it is based on probability theory (e.g. econometric model).’

 11. Model specific properties

−What are the main specific characteristics (strengths and weaknesses) of this model regarding the purpose of the recommendation?

−What are the new equations or essential equations?

−‘High level of technology detail.’

−‘It is assumed that demand can be derived from GDP with a linear formulation which could also be formulated nonlinearly.’

−‘For modeling shareholder strategies new equations are implemented and the objective function is extended.’

−[70], pp.8–25 lists several models with their specific strengths regarding the spatial or sectoral focus.

 12. Model interaction

−What is the input and output data of the model exercise?

−Does the model exercise include several models? If so, which data do these models exchange?

−Flow chart:

−Interaction matrix:

−[71], p.57 shows the interaction of several models.

 13. Model documentation

−Is there a complete documentation of the model available?

−Where are the units and symbols used in the equations documented?

−Is the model’s source code accessible?

−‘All used equations can be found in appendix A.’

−‘The open source model can be downloaded at…’

−See also [29, 33, 35–37]

−The Mobility Model in [72], pp.369–378 is fully documented in the Appendix but it is not available for download.

 14. Output data access

−Where can one find the numerical values (output) of the model?

−‘All output values can be found in appendix B.’

−[73] shows the key model output at the beginning of the study (p. 2) and some key values in the ‘Conclusions’ section (p. 74). The numerical values can be obtained from their model (freely available on their website).

 15. Model validation

−What kind of validation method is used?

−[74] and [18] distinguish between structural and behavioral validation.

−[75] illustrates 12 types of tests.

−[76], pp.130–134 devotes a chapter section to the model validation, including structural, parameter and behavior validation (cf. his Table 21).

 16. Post-processing

−Are the presented results directly taken from the models’ output or are they modified?

−Are additional assumptions applied for this modification?

−‘The model outputs are on a regional level, while the research questions aim at recommendations regarding a national level. Therefore, the output data is simply summated.’

−[77] project highway fuel demand based on their estimation of vehicle stock. Table 3 (pp. 160–161) shows that post-processing is undertaken for total vehicles for 2030.

 17. Sensitivity analyses

−How sensitive are the model results to parameter values variations?

−Univariate and multivariate sensitivity analyses: numerical, behavioral and policy sensitivity.

−Extreme conditions tests.

−[78], pp. 61–62 perform detailed sensitivity analyses for the total system cost of European Energy System.

 18. Robustness analyses

−Are the model results within the expected deviation compared to commensurable models?

−See also [55]

−‘All considered uncertainties concerning future developments can be found in the supplementary section.’

−[79] perform a robustness analysis of their model POWER against three other power system models.

 19. Results – recommend-dation - relationship

−How do specific recommendations correspond to the results (e.g. output value or interpretation)?

−How do results of the model exercise support normative recommendations with respect to the assumptions?

−How far can the recommendations be differentiated?

−[80], p. 3 recommends a global average fuel economy target based on emissions reductions and oil savings results.

 20. Uncertainty communication

−Which normative assumptions are considered for deriving recommendations?

−‘Based on the assumption that the expansion targets for renewable energies will still be the same for the next 20 years, we recommend to further stimulate investments into new renewable technologies by applying the following…’

−How reliable are the results due to the uncertainty of the assumptions and the modeling (e.g. system boundaries, model simplifications)?

−‘The specific oil-price development being used is highly uncertain. The only thing we know is that such a development is possible. And we also checked that all other assumptions being used to calculate the output are consistent with each other. Hence, what we calculated is possible.’

−‘If we use these simplifications it can be checked in a smaller model, that this simplification leads to an underestimation of….’

−With the summary for policy makers the IPCC [57] states comprehensively the uncertainties, for instance by differentiating the likelihood of the results and outcomes on p. 4.