Assessing the societal impacts of green demonstration homes: a Canadian case study

Abstract

Background

This article investigates the overall societal impacts of the REEP House for Sustainable Living (REEP House) in Kitchener, Canada.

Methods

Available information on green demonstration homes (GDHs) is reviewed to identify their goals, past assessment practices and their impacts on different measures ranging from energy consumption to behavioural changes. From this, the need for a multicriteria framework for evaluating GDHs is demonstrated. Drawing upon the GDH experience, the broader impact assessment literature, knowledge gained from community-focused recreational events and information from open eco-homes, such a framework is developed.

Results

This five category GDH multicriteria framework is then applied to the case of the REEP House. Using both technical data and social data, the results provide unique insights into GDH societal impacts across a variety of areas. The REEP House’s retrofits had significant impacts: reductions of electricity consumption by 41 %, of water consumption by 94 % and of gas consumption by 78 %. Its programming activities also showed noteworthy effects: regarding information distribution, 76 % of visitors felt they had received enough material to improve their own home; and with respect to the overall impact, more than 50 % stated that they were planning to return to the REEP House.

Conclusions

These results are compared with other GDHs’ experiences. In conclusion, lessons are drawn for all GDHs that wish to improve both their assessment procedures and their societal impacts. The limitations of this study are also identified.

Background

The REEP House

The city of Kitchener, Ontario, Canada, has a population of 220,000 people and is located approximately 80 km southwest of Toronto—Canada’s largest city. For years, the city government and many of its citizens have had a commitment to sustainability, as evidenced, for instance, through the presence and activities of both the City’s Environmental Committee and the many environmentally focused community groups based there. REEP Green Solutions (originally the Residential Energy Efficiency Project) is one such community group. Amongst its many activities, it has a green demonstration home (GDH)—the REEP House for Sustainable Living (henceforth called ‘the REEP House’)—with which it is attempting to change people’s behaviours through education.

Located in downtown Kitchener, the REEP House looks like other buildings in its neighbourhood—a simple home that was built in 1910. What is not immediately apparent however is that the REEP House is much more resource efficient than it was less than a decade ago [1]; it features an array of sustainability-minded retrofitting options, ranging from drain water heat recovery to a ground source heat pump and high efficiency lighting to spray foam insulated walls. After undergoing a deep retrofit beginning in 2009, the REEP House opened to the public for tours, workshops and other events in 2010.

REEP Green Solutions aims to highlight—to local homeowners—retrofits and technologies that can result in energy, water and cost savings. Through engaging with local homeowners using demonstrations, research and community campaigns with the physical house as the focal point, REEP Green Solutions staff seek to act as educators and facilitators to raise awareness and to empower actions. In doing so, REEP Green Solutions hopes to convince people to take steps to retrofit their own homes and to adopt sustainability practices by inspiring attitudinal and behavioural changes.

The research partnership

Mirroring a policy used by many not-for-profit organisations, entry to the REEP House is offered at no charge to the approximately 1000 people who pass through every year. Because of this, operating costs must be covered by grants and donations from various sponsors, as well as retained earnings from fee-for-service work. In order to justify their use of such funds, REEP Green Solutions and other not-for-profit organisations are increasingly being encouraged to determine the impacts of their operations.

However, limited resources and shifting agendas mean that sustained research efforts are, on the part of not-for-profit organisations, not always easily realised. As a result, some are beginning to recognise that collaborating with research institutions offers a unique opportunity for them to receive insights through detailed analyses; research institutions, for their part, get access to real-world data, thus allowing them to make timely contributions to society’s opportunities and challenges. It is in this spirit that the research partnership that resulted in this article proceeded.

As such, the purpose of this research was twofold: the first, to conduct a case study on the REEP House to uncover the overall impacts of its programmes and, the second, to explore impact assessment methods for GDHs and to determine whether these could be improved. We begin by reviewing the available information on GDHs to determine common goals and existing means of analysis. We complement this GDH-specific experience with insights from the broader impact assessment literature, knowledge gained from community focused recreational events and information from open eco-homes. With the resultant multicriteria framework we develop, we investigate the REEP House. Finally, we discuss what the REEP House and other GDHs could be doing to improve their impact assessments in the future.

The existing literature on green demonstration homes

We use the term green demonstration homes (GDHs) to refer to homes—either retrofitted or newly built—that are specially equipped with advanced environmental technologies and communication tools and/or strategies in order to show the general public how the housing stock can be made more sustainable. We thus define the term to exclude, amongst other things, houses that are being ‘lived in’ by residents and houses that showcase only one company’s products and/or services. Based on a search for ‘green demonstration homes’ and other similarly named projects, we identify 15 GDHs, along with their respective goals, in Table 1. The six goal categories in Table 1 are based on both the homes’ self-stated goals as articulated on organisation websites as well as themes from other information available online about each project.

Table 1 Goals of green demonstration home projects

A goal of 93 % of these homes is to educate and to raise awareness about more sustainable types of homes, 60 % want to show that such projects are just as feasible as standard building practices and just under one half of them are attempting to facilitate further adoption of green housing. In addition to the goals of the physical house, the organisations overseeing the operation may be working towards other goals that are not explicitly stated in available documents or are more specific than the categories in Table 1.

As previously discussed, organisations like REEP Green Solutions are being encouraged to show that they are achieving specific goals or offering value to their visitors and community. Having an effective way to show this, such as using a standard assessment framework, could make the task much easier for not-for-profit organisations that have little experience in programme impact assessment.

Early studies of GDHs focused primarily on performance of the new technologies being used in the building (e.g., [2]). Throughout the early 2000s, GDHs increased in popularity in the USA, Canada, Japan, the UK and Germany [3], but it was not until 2004 that Stoecklein and McKernon suggested a five tier methodology for GDH home evaluation [4]. Following this, we find that reports of individual GDHs might note one or more of the categories identified by Stoecklein and McKernon, though, as we note below, without explicitly referencing the same. Tylak’s evaluation of the effectiveness of the Piney Lakes Environmental Education Centre in Australia, for instance, used building performance and visitor numbers [5]. Additionally, Karol, Leggett and Siano evaluated the Subiaco Sustainable Demonstration Home in Australia in all categories except behaviour change [6]. The Utah House (USA), moreover, was declared an effective educational tool after Dietz, Mulford and Case reported significant increases in visitors’ self-reported knowledge [7]. We provide further examples of impact reporting in Table 2.

Table 2 The assessment categories in which GDHs have collected information

In summary, two key learnings can be extracted from the experience of the 15 GDHs. First, there is a large variety of criteria used in the respective impact assessments. And second, Stoecklein and McKernon’s research remains the only work that proposes an actual assessment framework for GDHs; moreover, no subsequent work has made use of it (or a modified version of it) to evaluate GDHs in a comprehensive way [4]. This limited experience with GDH assessment frameworks encourages us to review other literatures in order to develop a multicriteria framework for our investigation.

Other relevant literatures

We turn to three additional literatures to complement existing work on GDHs: first, general programme impact assessment; second, community focused recreational events; and third, open eco-homes.

The best practices for the general process of programme impact assessment have been refined in disciplines such as international development. Several guides outlining these best practices are available and can be utilised by those researching GDHs (e.g., [8–10]). Some of the key recommendations emerging from an analysis of this literature include the importance of using pre- and post-participation tests, randomised tests and control groups.

Turning to community-focused recreational events, public gatherings such as festivals, museum activities and sporting events each share with GDHs the goal of inspiring visitors to take action ‘at home’. Messages emerging from these literatures include the value of benchmarking and the potential utility of alternative methodologies [11–15]. With respect to the latter, Mair and Laing’s Transtheoretical Model, for example, provides a framework for the purpose of determining the pathway to behavioural change [16]. Again, such learnings can be applied to our work on GDHs.

Finally, the work on open eco-homes—that is, environmentally advanced homes that are made accessible to the general public by the owners/occupiers for a limited number of days—also offers insight. Berry et al. report that impact ‘is assessed in terms of visitor numbers and characteristics, experiences, and replication activities that emerge from engagement with an eco open home event’ (([17], p. 423); see, in a similar vein, [18]). Others have highlighted the importance of gathering more data about the subsequent actions of visitors, once they return home after their visit (e.g., ([19], p. 172; [20], p. 5)). We also take these insights forward.

Multicriteria framework

To further investigate the REEP House, we begin with the framework developed by Stoecklein and McKernon [4], which uses five categories within which ‘success’ should be achieved (Fig. 1). To this, we add key learnings pulled from our investigation into GDHs, as well as from our examination of the literatures relating to programme impact assessments, community-focused recreational events and open eco-homes. Accordingly, we offer a new multicriteria framework with five categories. Each of these is briefly elaborated below.

Fig. 1

Goals of green demonstration home projects (following [4])

Technology performance—are the technologies being monitored?

This involves monitoring and assessing the performance of both the individual technologies and the house as a whole. Data about energy consumption (disaggregated and whole house), air exchanges, water use and the interactions between the different technologies (for instance, do they support each other or are there redundancies?) are included in this category.

Educational programme performance—are the programmes at the house well organised and functioning properly?

This category assesses whether there is alignment between what organisers want visitors to learn at the house and what visitors are in reality coming away with. Are the right teaching techniques being used? Is the information accessible to the different types of visitors? What could be done to improve the visitors’ experiences?

Visitor numbers and other demographic information—how many people have interacted with the house and what other information is known about them?

This includes information like gender, age, reason for visiting and current type of home. The goal of getting these data is to find out if the home is attracting the desired target group (for example, those who have the most to learn about more sustainable living) and whether existing marketing strategies are effective in getting people to visit.

Attitude and value change—after visiting the house, did visitors change their attitude towards the topics presented in the house or their understanding of the value of sustainable housing?

Determining changes in visitors’ attitudes and values because of their visit to the GDH is the purpose of this category. It is important to note that, while still valuable information to know when assessing success, an attitude or value change in visitors does not necessarily translate into a change in behaviour [20].

Behaviour change—are visitors acting in new ways because of their visit to the home?

Success in this category is arguably the implicit goal for most GDH projects. Collecting these data is made easier when there is pre- and post-visit information that can be compared for differences. It can also illuminate the factors that may be preventing behaviour changes.

Table 2 uses the categories in this multicriteria framework to compare GDHs’ current evaluation practices, which range from minimal to extensive in each category. Many (67 %) GDHs undertake an evaluation of the number of people who come to visit the house, while the performance of the technology present is tracked by 53 %. The key take-away message however is that only 37 % of the total possible tracking actions have been taken, with a mean of 1.9 (of 5) actions completed per GDH.

Methods

In this case study, data were received from REEP Green Solutions staff. It is important to note that these data were initially collected prior to the development of the multicriteria framework described in this article. Instead, REEP Green Solutions staff used their understandings of GDH assessment to determine what information should be gathered between 1 January 2011 and 30 June 2013. These processes were not regularised within the organisation; as a result, at some times, short paper-based questionnaires were distributed to visitors and, with some, follow-up telephone calls or emails were undertaken approximately 3 months later. Consequently, different datasets often had different n values. In all instances, data were handled in a manner consistent with REEP Green Solutions’ privacy policies.

Because the original intent of the paper surveys was to improve their educational programmes, REEP Green Solutions introduced different surveys for the different tours and events they hosted. Some surveys had multiple iterations with only slight modifications – adding or removing one question, or slightly modifying wording in response to visitor feedback.

Our research involvement consisted of taking anonymous data from the paper surveys and organising them into spreadsheets, while all follow-up data had already been collected in a similar format. Data were then summarised further and questions were categorised based on the multicriteria framework introduced above.

Problems were encountered with the data provided as a result of not having control over aspects of the original question design and data collection methods. The surveys were not designed with pre-REEP House visit testing of visitors in mind, providing no baseline from which to measure changes in the visitors’ responses. It is also unclear in many of the follow-up questions whether the decisions made by homeowners were a direct result of contact with the REEP House or not. In addition, the inconsistent nature of the collected data means that individual question results are drawn from small numbers of respondents, limiting their ability to describe a broader population.

However, working with these data provides unique insight into the quality of information REEP Green Solutions uses and may also be suggestive of the state of information available to other GDHs more broadly.

Results

The original data collected from REEP Green Solutions were not organised in any particular way so the following results have been fit into the five categories of the GDH impact assessment multicriteria framework in an effort to give them a more meaningful structure.

Technology performance

The electricity monitoring system installed at the REEP House, along with access to utility data, allows for the tracking of whole house electrical, water and gas usage. Modelled results reveal that they have been reduced by 41, 94 and 78 % respectively resulting in a 54 % reduction in carbon emissions and cost savings of 85 % since the retrofit was completed [21].¹

Educational programme performance

Tours, workshops and other events at the REEP House are run by a combination of paid staff and volunteers who are trained to teach visitors about the many features of the house. While common themes form the core of most presentations, particular emphasis on various aspects of the tour is left to the staff member or volunteer.

Information about specific contractors (involved with the technologies and products shown in the house), and related government programmes, such as saveONenergy,² was provided to 30 and 40 % of visitors, respectively (Fig. 2). Of those who filled out surveys, 76 % felt they had received enough information to improve their home and property and 63 % enough to conserve water (Fig. 3).

Fig. 2

Questions relating to information distribution at the REEP House. Note: percentages do not add up to exactly 100 due to rounding

Fig. 3

Questions relating to educational programme performance at the REEP House. Note: percentages do not add up to exactly 100 due to rounding

There were no questions asked that tested visitors on the details of what they learned at the REEP House.

Visitor numbers and other demographic information

Figures 4 and 5 and Tables 3 and 4 present data regarding REEP House visitors who attended on either Wednesday evenings or Saturday afternoons (the usual opening times). The majority were in either the 20–29 years or the 50–59 years age groups (Fig. 4). Nearly three quarters of visitors were homeowners, over one half were planning to do, or were currently doing, home renovations, and over one half were simply interested in green living (Table 3). Visitors’ top motivations for coming to the REEP House included a desire for general knowledge about the house and because they were presently working on, or were planning to work on, an older house and were looking for ideas and suggestions (Fig. 5). Lastly, 34 % of visitors had had an ecoENERGY Evaluation³ conducted on their current home (Table 4).

Fig. 4

Age of visitors to the REEP House (n = 236). Note: percentages do not add up to exactly 100 due to rounding. There were also 19 blank responses (in addition to the 236 responses noted above)

Fig. 5

Motivation for visiting the REEP House (n = 48)

Table 3 Type of Visitors to the REEP House

Table 4 ecoENERGY evaluations (n = 171)

Attitude and value change

As seen in Table 5, of the visitors who had not had an energy evaluation done on their current home, 27 % said they intended to get one in the near future after visiting the REEP House. A large number of visitors (92 %) said they shared their experience with other people after visiting the REEP House, and over one half said they were planning to come back to the house for more information in the future (Table 6).

Table 5 REEP House visitors’ intention to get an energy evaluations (n = 124)

Table 6 Perception of the value of the REEP House (3 months after visit) (n = 37)

Behaviour change

In follow-up surveys, approximately 3 months after visiting the REEP House, 37 visitors were asked in a multiple-choice question what actions they had subsequently taken in their own homes (Table 7). Of the possible options, the most frequent action that people took or feature they added to their home was ‘Rain barrel or cistern’ and the least frequent was ‘Addition’ or ‘Grey water system’. Visitors were also asked which actions they were still planning on taking. The most commonly planned action was ‘Basement insulation’ and the least commonly planned actions were ‘Addition’, ‘Water heater’ or ‘Grey water system’. It would appear that visitors most often perform actions that require smaller time and/or capital investments, while the actions they are still planning to do after the 3 months are those that involve much more time and/or capital. Of course, it is also important to recognise that any correlation (an indication of action after a visit to the REEP House) does not necessarily mean causation [22].

Table 7 Actions taken by visitors within approximately 3 months of visiting the REEP house (n = 37)

Using a summary of the number of actions taken or planned by visitors, we see in Table 8 that over one half of the visitors to the REEP House took some kind of action after their visit and almost one half of the visitors were still planning to take action when surveyed; many had done both.

Table 8 Summary of Number of Actions Taken and Planned by REEP House Visitors (following Table 7) (n = 37)

Discussion

In light of the application of the multicriteria framework to the case of the REEP House, three important findings emerge.

First, we can reflect upon the findings from this particular case study and compare them to the previous experiences of other GDHs. As noted in Table 2, while not every GDH has been evaluated on every criterion in our framework, it is still the case that some cross-comparisons can be made. The Florida House Learning Center, for instance, reported impressive educational programme performance: ‘… 100 % [of first-time visitors] gained new knowledge, 99 % intended to change their behaviors …’ ([23], p. 233). Turning to visitor numbers and other demographics, the Utah House reported that its visitors were ‘predominantly Caucasian (92.9 %), aged 45–64 (55.1 %), female (70.1 %) and highly educated [most had some college-level education]’ ([7], p. 1709).

More difficult to find is actual change after the visit to the GDH—the change in either attitudes and values or behaviours. (This is not surprising, given that only two of the 15 GDHs reviewed had that as an assessment criterion (see Table 2).) Nevertheless, the Christchurch Energy Efficiency Show Home, for instance, found that ‘more than 50 % of people do act after getting advice from the Show Home. This relates especially to under floor insulation and double glazing’ [24]. The paucity of data—and the range of methods used amongst those that do investigate this factor (as well as sample sizes secured, not to mention confounding factors across different studies)—means that any systematic comparison would not have statistical significance.

Our second finding is that we executed the framework successfully—we walked through all five categories, and it seemed to be a useful way of organising data.

Of the five categories in the multicriteria framework, this case study found both that the REEP House can be viewed as a leader amongst these GDHs and that improvement and elaboration of the assessment techniques are still possible. More specifically, by taking steps to assess at least part of each of the five categories of the multicriteria framework, the REEP House fares far better than all of the GDHs listed in Table 2. Similarly, however, there is still scope for improvement in terms of both the depth and the breadth of the analysis. Table 9 describes, for each category of the multicriteria framework, what the data collected by the REEP House tell us, what could have been done—and should be done—to offer better insights as well as general recommendations for any GDH that would like to undertake assessment in that category in the future.

Table 9 Multicriteria framework: past and prospects

In addition to the category-specific suggestions in Table 9, there are three more general actions all GDHs could do to improve the ways in which they assess their impacts. First, GDHs could encourage visitors via email to answer a follow-up survey after their visit about their experience at the house. Follow-up data could be automatically linked to pre-visit data using email addresses, while other possible identifying information (income, age, house type) would be kept separate.⁴

Second, GDHs could create a basic survey that every visitor would get, which asks for demographic information and attitudes, values and motivations. In addition to this, questions unique to the event type being attended (tour, workshop) that fall into certain categories in the multicriteria framework could be added.

And third, GDHs could follow good research practices by being intentional when designing surveys and questionnaires. They could ensure that the questions asked will provide information that achieves the appropriate breadth and depth of understanding.

We now turn to the third important finding emerging from this investigation.

This study has not only served to contribute to understanding about Green Demonstration Homes in ways described above, but it has also added to discussions in related conceptual areas, in sustainable residential buildings and in sustainable communities more generally.

First, this case study—which draws upon literatures surrounding general programme impact assessment, community focused recreational events and open eco-homes—can similarly contribute to those same areas. As but one example, a recent study examining the extent to which sporting events encouraged attendees to participate in sports themselves noted methodological challenges around self-reporting, sample size and causality [25]. It is thus clear that these distinct literatures could potentially benefit from cross-communication.

Second, while it is widely recognised that the development of better buildings is a major challenge for a range of sustainability issues—not least of all, global climate change—many have noted that commercial buildings often receive more attention than residential buildings (e.g., [26], p. 388). Indeed, this is notwithstanding the fact that the latter is responsible for the majority of the building stock and associated environmental impacts (e.g., [27]). Consequently, this case study, by prompting closer consideration of residential building performance and catalysts for change, adds to this important discussion.

And third, local innovations (sometimes called ‘experiments’) as part of community-led sustainability endeavours are increasing in number around the world; similarly, they are attracting more attention from policy-makers (e.g., [28]). Thus, the details provided in this case study—about a grassroots organisation working for a sustainability transition—contribute to these conversations.

Conclusions

Based on the findings of this case study and other current research on green demonstration homes (GDHs), it is not always entirely clear whether such projects are having all of the impacts they desire. Most GDHs are not evaluating all of the possible elements that they could be (which should, of course, also be a function of their own particular goals). Consequently, no systematic conclusion about either their impact upon visitors or upon their effectiveness in a broader sense can be made at this time. Indeed, even with respect to particular conclusions for the REEP House, our ability to reach conclusions is necessarily limited by two key realities: first, the fact that the multicriteria framework is not being implemented within the REEP House’s procedures until after the data collection stage and, second, the fact that data monitoring regarding long-term behavioural changes is not being undertaken systematically.

Future assessments of GDHs could use the multicriteria framework as outlined in this article. More work could also be done to refine the questions being asked in each of the framework categories or to improve the framework as a whole. Investigation into how a GDH supports broader community programming—because it could be that impacts are felt elsewhere in an organisation’s portfolio of activities—may be worth pursuing. In any case, the results of this study represent one step towards the implementation of improvements such as these, as they bring awareness to the issue at hand—namely, the current state of evaluation of GDHs and the potential for improvement in the same. If the results of future GDH impact assessments are positive, as elements of the REEP House case study reveal, then they could be used to justify the existence and continued development of GDH projects. If not, however, such organisations would need to examine other ways of achieving their goals, perhaps by using GDHs in new ways.

Abbreviation

GDH, green demonstration home