Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (2024)

Prepared by:
Office of Audit and Evaluation
National Research Council Canada

November14, 2016

Acknowledgements

The Office of Audit and Evaluation would like to gratefully acknowledge the management and staff of the Herzberg Astronomy and Astrophysics ( HAA ) Portfolio for their support and contribution to this evaluation. The evaluation team would also like to express its gratitude to members of the Evaluation Advisory Committee for their guidance and advice. Finally, the evaluation team would like to thank HAA partners and other stakeholders who provided rich information and insights in support of this evaluation.

Executive summary

Portfolio description

NRC is mandated by the NRC Act to "operate and administer any astronomical observatories established or maintained by the Government of Canada". NRC - HAA is the portfolio responsible for fulfilling this mandate, including Canada's participation in all current and future offshore facilities. HAA provides financial contributions and designs/develops instruments to support the administration and operation of observatories supported by Canada.

For the period covered by the evaluation, HAA directly supported two national observatories located in British Columbia, the Dominion Astrophysical Observatory ( DAO ) and the Dominion Radio Astrophysical Observatory ( DRAO ), as well as six offshore observatories: Canada-France-Hawaii Telescope ( CFHT ), Gemini, James Clerk Maxwell Telescope ( JCMT ), Atacama Large Millimetre/submillimetre Array ( ALMA ), Thirty Meter Telescope ( TMT ), and Square Kilometre Array ( SKA ).

International Telescope Agreements Gs&Cs expenditures

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (1)

Long description of International Telescope Agreements Gs&Cs expenditures

This figure is a horizontal bar chart comparing the grants and contributions ( G&Cs ) expenditures associated with the International Telescope Agreements. The figure shows an increase in expenditures from $10.15million in FY 2011-12to $15.47million in FY 2015-16.

YearMillions of dollars
2011-1210.15
2012-1310.67
2013-1410.34
2014-1512.15
2015-1615.47

Scope and methodology

The evaluation examined HAA Portfolio and program spending between FY 2011-12and FY 2015-16. A complete and in-depth assessment of the performance of NRC 's internal services was not within the scope of the evaluation. In accordance with the TBS Policy on Evaluation, the evaluation covered the core issues of relevance and performance, including effectiveness and efficiency and economy.

Challenges and limitations

  • The level of client satisfaction with CADC services was not assessed.
  • NRC Corporate KPI used to assess efficiency does not include corporate administrative costs.

Overall evaluation findings

The evaluation found HAA to be relevant to NRC and federal government mandates. Further, HAA meets the needs of the Canadian astronomy community as the Portfolio ensures that its activities and funding priorities are aligned with the Canadian Astronomical Society's ( CASCA ) Long Range Plan 2010( LRP2010 ). Canadian astronomers report that HAA plays a critical role in representing the interests of the Canadian astronomy community internationally through its involvement in existing and future observatories. Additionally, HAA 's facilities at DAO and DRAO are unique in Canada, providing core capabilities and expertise in the development of observatory instrumentation.

HAA has performed very well and has had significant impacts by supporting Canada's offshore observatories as well as other telescopes around the world. HAA support has been primarily focused on financial support and the development of world-class instruments and technologies. HAA has collaborated with industry on instrumentation projects, which have resulted in positive impacts for the partnering companies. Most stakeholders perceive HAA as vital in supporting Canada's performance in the field of astronomy.

The evaluation also found that HAA has encountered operational challenges which may impact HAA 's ability to meet its objectives. Challenges included procurement delays, lengthening timelines for staffing actions, and limitations in IT desktop support. Further, the evaluation found that the transfer of HAA 's IT infrastructure to Shared Services Canada ( SSC ) has had major impacts on the Portfolio's ability to plan, implement and acquire new IT equipment and networking capacity. Despite these challenges, the Portfolio consistently delivered observatory instruments on time, on budget and according to required specifications.

The overall assessment of HAA 's performance by evaluation issue is presented in the table below.

Assessment of HAA 's performance
IssueAssessmentAssociated recommendations
Relevance
Alignment with the needs of Canadian astronomersLow area of concern1
Appropriateness of current delivery modelMeets expectationsNone
Alignment with federal priorities and NRC strategic objectivesLow area of concern2, 3
Alignment with federal roles and responsibilitiesMeets expectationsNone
Performance
Success in providing Canadian astronomers with access to national and international telescopes and science data productsLow area of concern4
Scientific impacts of international telescopes administered and operated by HAA Meets expectationsNone
Contribution to the development of novel instruments and technologiesMeets expectationsNone
Scientific and industry impacts of instruments and technologies developed by HAA Meets expectationsNone
Support for Canadian universities by training students and post-doctoral researchersMeets expectationsNone
Contribution to strengthening Canada's position in the field of astronomyMeets expectationsNone
Efficiency and economy
HAA is administered in a cost-effective and efficient mannerMeets expectationsNone
Key risks and factors influencing the cost-effective and efficient use of resources by HAA Management attention required5

Summary Finding

Some foundational processes are in place to support major capital planning and execution.

Challenges in contingency definition, cost estimate development, risk management, and execution strategy, have resulted in delayed, off-scope, and/or over-budget projects. Inconsistent approaches and interpretations of processes stemming from the lack of a single source of guidance, hampered project teams across planning and execution phases. Best practices and lessons learned are not being shared within the community to drive improvement.

Targeted guidance is required to better support the development of information related to project estimation, contingency planning, and risk management.

Recommendation 1:

The Vice-President's Office ( VPO ) of Emerging Technologies– National Infrastructure and Future Technologies, in consultation with HAA , should consider establishing a formal consultation process to allow the Canadian astronomy community to provide strategic advice on the Portfolio's scientific activities and priorities.

Management response:

Accepted. NRC will investigate appropriate mechanisms for obtaining advice on a regular basis from the astronomy community and will develop a proposed process early in FY 2017-18.

Recommendation 2:

HAA should continue to work with partners to expand outreach and awareness activities.

Management response:

Accepted. HAA will continue to work in concert with partners to expand outreach activities. A new model to optimise use of the Centre of the Universe will be explored.

Recommendation 3:

HAA , in collaboration with other NRC Portfolios and NRC Business Management Support ( BMS ), should explore opportunities to leverage competencies and astronomy-related technologies.

Management response:

Accepted. HAA with support from the VPO of Emerging Technologies– National Infrastructure and Future Technologies and BMS will engage into discussions with the broader NRC community and ensure that competencies and intellectual property held at HAA are known and shared.

Recommendation 4:

HAA should identify the reasons why Canadian PI s are not allocated observing time commensurate with Canada's financial support to ALMA operations and take the appropriate measures identified, as needed.

Management response:

Accepted. HAA will work with the Canadian astronomy community and with the National Science Foundation to understand why the allocation of ALMA time to Canadian astronomers within the allocation of ALMA time to the North American Consortium has not been proportional to Canada's share of the North American Contributions and will attempt work to remove any impediments to Canadian access.

Recommendation 5:

HAA , in collaboration with KITS , should continue to undertake efforts to find a solution related to the IT equipment upgrades and high speed networking capacity issues at the CADC .

Management Response:

Accepted. NRC will continue to work to find a solution to ensure that CADC services to clients are not impacted.

Acronyms

ACURA Association of Canadian Universities for Research in Astronomy
AO Adaptive optics
ATP Astronomy Technology Program
CADC Canadian Astronomy Data Centre
CanTAC Canadian Time Allocation Committee
CANFAR Canadian Advanced Network for Astronomical Research
CASCA Canadian Astronomical Society– Société canadienne d'astronomie
CFI Canada Foundation for Innovation
CSA Canadian Space Agency
CSP Central Signal Processor
DAO Dominion Astrophysical Observatory
DRAO Dominion Radio Astrophysical Observatory
ESPaDOnS Echelle SpectroPolarimetric Device for the Observation of Stars
GPI Gemini Planet Imager
HAA Herzberg Astronomy and Astrophysics
HIA Herzberg Institute for Astrophysics
HPC High-performance computing
ITA International Telescope Agreements
KITS Knowledge and Information Technology Services
KPI Key Performance Indicator
LNA Low noise amplifier
LRP2000 Long Range Plan 2000
LRP2010 Long Range Plan 2010
MAG Millimetre Astronomy Group
MTR Mid-Term Review of the Long Range Plan 2010
NASA National Aeronautics and Space Administration
NFIRAOS Narrow Field Infrared Adaptive Optics System
NRAO National Radio Astronomy Observatory
NRC National Research Council
NRCan Natural Resources Canada
OAP Optical Astronomy Program
PAF Phased array feed
PAndAS Pan-Andromeda Archeological Survey
RAP Radio Astronomy Program
SSC Shared Services Canada
WIDAR Wideband Interferometric Digital Architecture
Telescopes
AAT Anglo-Australian Telescope
ALMA Atacama Large Millimeter/submillimeter Array
CFHT Canada-France-Hawaii Telescope
CHIME Canadian Hydrogen Intensity Mapping Experiment
ESO3p6 European Southern Observatory 3.6m Telescope
EVLA Expanded Very Large Array
GBT Green Bank Telescope
HET Hobby-Eberly Telescope
HST Hubble Space Telescope
IRAM-30m Institut de radioastronomie millimétrique
IRTF Infrared Telescope Facility
JCMT James Clerk Maxwell Telescope
JVLA Jansky Very Large Array
JWST James Webb Space Telescope
LBT Large Binocular Telescope
MMT Massive Monolithic Telescope
NTT New Technology Telescope
SALT Southern African Large Telescope
SKA Square Kilometre Array
SOAR Southern Astrophysical Research Observatory
TMT Thirty Meter Telescope
TNG Telescopio Nazionale Galileo
UKIRT United Kingdom Infrared Telescope
VLT Very Large Telescope

1.0Introduction

This report presents the results of the 2015-16evaluation of the National Research Council ( NRC ) Herzberg Astronomy and Astrophysics ( HAA ) Portfolio. The evaluation of HAA was conducted in fiscal year 2016-17. The timing of the evaluation is bound by the legal requirements of the Financial Administration Act, which requires completion of the evaluation of HAA 's contribution to international observatories by November 2016.

The evaluation examined HAA Portfolio and program spending over a five year period, FY 2011-12to FY 2015-16. However, when exploring the scientific and industrial impacts of the instrumentation developed by HAA , the evaluation also considered activities conducted prior to FY 2011-12.

The evaluation was led by an independent evaluation team from NRC 's Office of Audit and Evaluation ( OAE ) and assessed the core issues of the 2009Treasury Board Policy on Evaluation, including relevance and performance (effectiveness and efficiency and economy). The questions addressed by the evaluation were developed following a review of key program documents and in consultation with HAA management and the VP , Emerging Technologies.

The evaluation methodology used multiple lines of evidence and complementary research methods as a means to enhance the reliability and validity of the information and data collected. The methodology included the following lines of evidence:

  • Document and literature review
  • Analysis of financial, administrative and performance data
  • Survey of Canadian telescope users ( n =95)
  • Key informant interviews (internal interviewees n=13, external interviewees n =18)
  • Case studies ( n =4, interviews n=18)

A more detailed description of the study methodology and its limitations is provided in AppendixB: Methodology.

Section 2 of this report provides a short profile of HAA . Sections 3 and 4 present the evaluation study's findings organized by broad evaluation question (relevance and performance) along with associated recommendations. Section 5 presents a brief conclusion drawn from the evaluation, while Section 6 lays out management's response to these recommendations and the actions that will result.

2.0Profile of HAA

In the late 1960s, NRC was mandated, through the NRC Act, to "operate and administer any astronomical observatories established or maintained by the Government of Canada" (Justice Canada, 2009). In 1974, the interpretation of NRC 's mandate was broadened to include Canadian participation in offshore observatories through an agreement with France and the University of Hawaii to establish the Canada-France-Hawaii Telescope Corporation. Restructured in 2012, HAA continues to operate the programs conducted by its predecessor, the Herzberg Institute of Astrophysics ( HIA ), that fulfill the mandate set forth by the NRC Act, including Canada's participation in all current and future offshore facilities.

While the NRC Act underpins the NRC 's involvement in astronomical research, HAA 's activities are primarily guided by the Canadian Astronomical Society's ( CASCA ) Long Range Plan 2010( LRP2010 ), a decadal plan outlining the broad goals and directions of astronomical and astrophysical research in Canada. LRP2010 , as a continuation of LRP2000 , puts forward a plan to "maintain Canada's position in the upper echelon of international astronomy" by securing Canadian participation in existing and future "world-leading" ground- and space-based observatories ( CASCA , 2010, pp. 1–9).

For the period covered by the evaluation, HAA directly supported two national observatories located in British Columbia and six offshore observatories, presented in Table1. Descriptions of the offshore facilities are available in AppendixC: Overview of international telescopes supported by Canada.

Table1: Telescopes supported by HAA between 2011-12and 2015-16
TelescopeLocation
National Observatories
Dominion Astrophysical Observatory ( DAO )Victoria, British Columbia
Dominion Radio Astrophysical Observatory ( DRAO )Penticton, British Columbia
Offshore Observatories
Canada-France-Hawaii Telescope ( CFHT )Summit of Maunakea, Hawaii
Gemini(North) summit of Maunakea, Hawaii (South) Cerro Pachón, Chile
James Clerk Maxwell Telescope ( JCMT )Table note *Summit of Maunakea, Hawaii
Atacama Large Millimeter/submillimeter Array ( ALMA )Chajnantor Plateau, Atacama Desert, Chile
Thirty Meter Telescope ( TMT )– under constructionSummit of Maunakea, Hawaii (planned)
Square Kilometre Array ( SKA )Table note **pre-constructionAustralia and South Africa (planned)

Table notes

Table note *

NRC withdrew from the JCMT partnership and ceased its financial contributions on September 30, 2014.

Return to table note * referrer

Table note **

While Canada is committed to the pre-construction phase for SKA , the federal government has not yet decided on participation in the construction and operation phases for SKA .

Return to table note ** referrer

A logic model, developed as part of evaluation planning, details the activities, outputs and intended outcomes of the HAA Portfolio (AppendixA: HAA Logic Model). A brief overview of HAA , including its programs, clients and financial and human resources, follows.

2.1Portfolio and program structure

To fulfill its mandate and to advance the LRP2010 , HAA operates three programs: the Optical Astronomy Program ( OAP ), the Radio Astronomy Program ( RAP ) and the Astronomy Technology Program ( ATP ). While the HAA programs were created as distinct components in response to the restructuring of NRC , they are highly integrated within the Portfolio and many program activities are intertwined. Each program is briefly described below.

  • Optical Astronomy Program ( OAP ): The OAP supports and facilitates access to optical, ultraviolet and infrared telescopes for Canadian astronomers. OAP staff provide in-kind and contracted support for the operation of Canada's observatories and act as the interface between the Canadian astronomy community and the CFHT and GeminiFootnote 1 observatories. The OAP also operates the Canadian Astronomy Data Centre ( CADC ), Canada's astronomy data management and archiving facility, which provides data handling and information technology services to Canadian and international astronomers.
  • Radio Astronomy Program ( RAP ): The RAP is responsible for supporting and facilitating Canadian access to millimetre/submillimetre and centimetre observatories. The RAP is comprised of two groups: the Millimetre Astronomy Group ( MAG ) and the Dominion Radio Astrophysical Observatory ( DRAO ). The MAG acts as the interface between the Canadian astronomy community and ALMA , Canada's only offshore submillimetre observatory. The DRAO provides science and technical services to support Canada's national and international radio telescopes.
  • Astronomy Technology Program ( ATP ): The ATP is responsible for the design and development of instruments and observatory infrastructure for the national and international telescope facilities supported by Canada. Through multifunctional teams, the ATP provides technical and advisory services, including: project management, system engineering, adaptive optics ( AO ), mechanical/electrical/optical/radio engineering, instrumentation science, millimetre technology, software and controls.

2.2Clients and stakeholders

As per its mandate to operate and administer Canada's telescopes, the main clients of the HAA Portfolio are Canadian professional astronomers and university students who access the telescopes to conduct research. In order to meet the needs of these researchers, HAA partners with national and international collaborators. As such, unlike other NRC Portfolios, HAA does not serve a specific industry segment that pays to access R&D services. HAA 's main client and stakeholder groups include the following:

  • Universities and academia: Canadian astronomers, largely university faculty, graduate students and postdoctoral researchers, are HAA 's primary clients. By partnering with other countries for the construction, commissioning and operation of observatories, Canada, through NRC , provides Canadian astronomers with access to international telescopes. Additionally, HAA consults and collaborates with the Association of Canadian Universities for Research in Astronomy ( ACURA ) and CASCA to ensure its activities are aligned with the needs of the astronomy community.
  • Observatory partners: HAA , on behalf of the Canadian astronomy community, collaborates with other countries and international organizations to establish partnerships for building, operating and managing international observatories.
  • Industry: HAA works with Canadian and international companies in the development of major instrumentation and development projects. Typically, companies are contracted for the procurement of non-standard parts and assemblies.
  • Canadian Space Agency ( CSA ): HAA collaborates with the CSA , providing scientific and technical resources to support the CSA and its industry contractors in international space-based astronomy projects.

2.3Portfolio resources

2.3.1Financial resources

Table2 presents the overall expenditures and funding sources for HAA since FY 2012-13. During this period, HAA 's programs generated a total of $9.96million in revenues. More than a third of these revenues (37percent) were produced in FY 2015-16, and are largely attributable to the Astronomy Technology Program ( ATP ) ($2.5million) from projects for MeerKAT, Gemini High-Resolution Optical Spectrograph ( GHOST ) and CFHT SpectroPolarimètre Infra-Rouge ( SPIRou ).

Table2: HAA financial profile (in $ millions)
Fiscal years
2012-132013-142014-152015-16
Expenditures
Salaries14.0814.5813.8314.08
Direct operations4.504.735.186.89
Indirect operations & facilities0.580.740.550.81
Total19.1520.0519.5621.78
Sources of funds
Revenues2.201.672.433.66
Appropriations16.9518.3817.1317.77
TMT funding- - - 0.35
Total19.1520.0519.5621.78
Expenditures funded by other sources
G&Cs 10.6711.3412.1513.55
G&Cs TMT - - - 1.91
Minor capital0.210.450.490.61

Source: HAA administrative data.

Note: Financial information for FY 2011-12is not available due to changes to the NRC accounting structure as part of NRC 's transformation.

2.3.2Grants and contributions in support of International Telescope Agreements

Table3 shows expenditures for each of the observatories in which Canada is a partner.

Table3: International Telescope Agreements G&C expenditures (in $ millions)
ObservatoriesFiscal years
2011-122012-132013-142014-152015-16
In operation
CFHT 3.113.123.323.574.25
Gemini5.565.676.116.597.33
JCMT 0.700.120.130.06-
ALMA 0.791.431.401.571.58
Under construction
TMT Table note *----1.91
SKA -0.330.380.360.39
Total10.1510.6711.3412.1515.47

Source: Financial data provided by NRC Finance Branch.

Table notes

Table note *

While Canada made an initial $1.9million contribution to the TMT project in 2015, the bulk of the TMT contribution is design work done by HAA staff directly or through contracts with industry to deliver HAA 's "workshare" contribution to the TMT project ( i.e. , "in-kind" contribution and not a cash payment).

Return to table note * referrer

Expenditures for the observatories in which Canada is a partner increased 52percent between FY 2011-12and FY 2015-16, with one exception. Canada reduced JCMT support and eventually withdrew from the partnership in September 2014. JCMT funding was then redirected to support ALMA operations, as recommended by LRP2000 .

Increases in financial contributions for observatory operations are influenced by three principal factors:

  1. Exchange rate fluctuations

    Canada's financial contributions to offshore observatories are adjusted in accordance with the value of the Canadian dollar relative to the currency specified in a given observatory's governing agreement. Fluctuations in foreign currency and exchange rates result in uncertainty when budgeting and estimating operations costs for Canada's offshore facilities. Based on findings from the interviews, this was the main contributing factor for the variations observed for the G&C funding envelope.

  2. Changes in observatory budgets

    Funding requirements for HAA are based on obligations arising from International Telescope Agreements ( ITA ). The budgets of the international observatories are approved by the respective observatory Boards, of which Canada is a member. These budgets fluctuate from year to year.

    To manage the two issues described above, G&Cs reflecting approximately 90percent of the anticipated annual cost of $10million are set and then adjustments are made accordingly via supplementary estimates and internal transfersFootnote 2.

  3. Increased funding provided for the construction and commissioning of TMT

    ITA expenditures increased 27percent in FY 2015-16($3.3million) as a result of the commencement of annual in-kind contributions to TMT . In its 2015Budget, the federal government confirmed its commitment to allocate $243.2million over 9years for the construction and commissioning of TMT , which includes $218.2million through NRC and $25million from the Canada Foundation for Innovation ( CFI ). This investment is expected to secure an estimated 15percent stake in the $1.5billion TMT project. The first financial contribution to the project ($1.9million) was recorded in FY 2015-16. In line with Canada's 15percent share in the project, NRC is expected to provide an estimated $6million per year to TMT 's annual operating costs once construction is complete, which is expected for FY 2026-27.

2.3.3Human resources

Overall, the number of individuals employed by HAA decreased by 19percent over the last five years, from 170to 137, mainly as a result of the centralization of common support services (within NRC Common Services and Shared Services Canada ( SSC )). In particular, there were reductions in Technical Support Staff (Computer Systems Administration and Technical Category) and Administrative Staff (Administrative Support). However, it should be noted that the number of core research staff, including Research Council Officers, Research Officers, and Research Associates, increased slightly during the evaluation period from 77in 2012to 81in 2016.

3.0Relevance

The relevance of HAA was examined through four evaluation issues: HAA 's alignment with the needs of Canadian astronomers (Section 3.1); the appropriateness of its delivery model for the administration and operation of Canada's telescopes to meet the needs of Canadian astronomers (Section 3.2); its alignment with the priorities of the federal government and NRC 's strategic objectives (Section 3.3); and its alignment with federal roles and responsibilities (Section 3.4).

3.1Alignment with the needs of Canadian astronomers

The evaluation assessed the extent to which HAA is aligned with the needs of Canadian astronomers. More specifically, the evaluation examined: HAA 's consultations with the Canadian astronomy community to ascertain its needs; the rationale for investing in national and international telescopes; the demand for accessing Canada's offshore observatories; and the Canadian astronomy community's needs in terms of access to telescope data provided through the CADC .

3.1.1Consultations with the Canadian and international astronomy communities

Needs of the Canadian astronomy community identified through consultations

Key Finding 1: HAA 's activities and priorities are closely aligned with the needs of the Canadian astronomy community.

To ensure its activities and investment priorities are aligned with the needs of the Canadian astronomy community, HAA takes guidance from CASCA 's LRP2010 and regularly consults the Canadian and international astronomy communities. The LRP2010 Panel, commissioned by CASCA with support from National Science and Engineering Research Council ( NSERC ), NRC , CSA , CFI and ACURA , was mandated to engage the research community and identify investment priorities in astronomy and astrophysics. Canadian astronomer interviewees confirmed that HAA 's activities are aligned with the recommendations and objectives outlined in LRP2010 . A Mid-Term Review ( MTR ) of the LRP was undertaken in 2015which reconfirmed the directions outlined in the 2010plan.

LRP2010 outlined nine recommendations that are relevant to HAA 's role in supporting Canada's involvement in ground-based instruments and observatories. Although HAA is not required to implement the recommendations, those recommendations which could be addressed (7in total), have been. This suggests that the Portfolio is aligned with and has been responsive to the needs of the Canadian astronomy community.

Further, HAA regularly consults the Canadian astronomy community, through its involvement in various committees. HAA staff members are involved in the Canadian Time Allocation Committee ( CanTAC ) and various CASCA committees. Currently, CanTAC has 11members from Canadian and American universities and two members from HAA . In addition, as of July 2016, HAA staff members participate in nine out of 11 CASCA committees. Involvement in these national astronomical organizations allows HAA to interact directly with the Canadian astronomy community and gain a better understanding of its needs.

Perspectives on the need for an HAA advisory committee

Key Finding 2: There is a need to establish a formal consultation process, such as an external advisory committee, to allow the Canadian astronomy community to provide advice on HAA 's activities and strategic priorities.

Prior to the NRC Transformation, NRC institutes each had an advisory board which provided strategic advice on institute activities and priorities. The NRC Herzberg Advisory Board ( HAB ) provided strategic advice to the Director General ( DG )Footnote 3 on HIA 's programs and activities. More specifically, the board provided:

  • expert guidance on trends and developments of relevance to HIA
  • feedback on research, technology and innovation issues
  • advice to management on strategic directions and priorities
  • oversight on the alignment of HIA 's programs with the priorities of the LRP
  • support to HIA in fulfilling its national mandate

The HAB was dissolved in 2011, along with the other NRC Institute advisory boards. As a result, the astronomy community in Canada does not currently have a formal mechanism through which it can provide input to the Portfolio on priorities and objectives related to astronomy. The Mid-Term Review of the LRP2010 highlighted this issue and recommended that the NRC Herzberg Advisory Board, constituted of senior representatives of academia and industry, be restored to advise HAA on its activities and programs and to represent the astronomy community's interests more effectively ( CASCA , 2016, p. 99).

Some internal interviewees and most external interviewees expressed support for the creation of a formal consultation process or mechanism, such as an external advisory committee, which would allow the Canadian astronomy community to provide strategic input on HAA 's activities. However, key informants acknowledged that HAA does conduct informal consultations with the community.

Since NRC manages the national astronomy program, key informants noted that HAA 's activities and strategic priorities should be determined in close consultation with Canadian astronomers, the Portfolio's core client community. A few key informants indicated that there is a need for more frequent consultations between HAA and the astronomy community ( e.g. , annually) given that the LRP and MTR processes are only every five years. According to key informants, the creation of an advisory committee has potential benefits, including:

  • offering a level of transparency and oversight of HAA 's activities
  • fostering trust between HAA and its client community
  • ensuring more frequent formal consultations between the Portfolio and the Canadian astronomy community with regard to priorities ( i.e. , annually as opposed to every five years)
  • informing the community of ongoing challenges faced by HAA

On the other hand, a few key informants reported that creating an external advisory committee would likely result in additional financial costs ( e.g. , travel and accommodations for committee members) that may be incurred in HAA 's operating budget. The creation of a new committee would also result in additional time commitments for HAA and the astronomy community, such as additional reporting requirements and meeting preparation. A few key informants noted that if a new consultation process was to be instituted, there may be a need to manage community expectations as it relates to the level of the committee's influence on HAA 's activities. While the committee would provide strategic advice to HAA , it would not have direct authority over program activities ( i.e. , HAA is responsible for delivering NRC 's mandate).

Recommendation 1: The Vice-President's Office ( VPO ) of Emerging Technologies– National Infrastructure and Future Technologies, in consultation with HAA , should consider establishing a formal consultation process to allow the Canadian astronomy community to provide strategic advice on the Portfolio's scientific activities and priorities.

Consultations with the international astronomy community

In addition to its consultations with the Canadian astronomy community, HAA is engaged in consultations with international partners, through its participation in the governance of international observatories. As of July 2016, HAA is represented at five international observatories: CFHT , Gemini, ALMA , TMT and SKA . Data indicates that a total of fourteen HAA representatives are members of twelve different boards and committees across the five observatories. Most importantly, HAA had at least one representative on the board and science advisory committee at each of the five observatories. This involvement allows HAA to gain a better understanding of the needs and requirements of observatory partners and to represent the interests of the Canadian astronomy community at the observatory level.

3.1.2Need for international and national observatories

Key Finding 3: There is a need for HAA to support administration and operation of telescopes to ensure that Canadian astronomers have access to world-class facilities.

Need for international facilities

The evaluation confirmed the findings from the 2011evaluation and the 2011Hickling Arthur Low "Astronomy in Canada" study (HAL, 2011) that identified the importance of Canada's contribution to international telescopes for:

  • Cost-sharing between countries for increasingly expensive and complex research facilities and instruments: The scale, cost and technological complexity of building large telescopes are driving increased internationalization of observatory instrumentation and infrastructure development. As a result, Canada and other countries rely on international partnerships, which normally require financial and in-kind contributions, to operate and manage observatories.
  • Providing contracting opportunities for Canadian industry and universities: For example, Canada's investment in SKA and TMT will likely lead to new commercial benefits for Canadian companies and will result in opportunities to attract and retain highly skilled workers into Canada's workforce ( CASCA , 2016, p. 46; Doyletech 2014). Contracting opportunities also exist with CFHT , Gemini and ALMA .
  • "Pay to play" principle: Through its financial and in-kind contributions, Canada is guaranteed observing time at the observatory and is given the opportunity to participate in the facility's governance structure. This provides Canada with the ability to promote the interests of the Canadian astronomy community and to have some degree of influence over the scientific direction of the observatory.
  • Accessing observatories to conduct competitive research: Canadian astronomers need access to world-class facilities to conduct competitive research, as highlighted by the LRP2010 ( CASCA , 2010).

According to key informants, two additional reasons for supporting international observatories are:

  • Collaborations: Observatory partnerships are multilateral in nature and involve partners from different countries. This enables researchers to facilitate knowledge and personnel exchange through research projects.
  • Training of students: Access to international observatories is key to training students and providing them with experience in the field ( e.g. , preparing a proposal and conducting their own observations).
Need for national facilities

Supporting and operating Canada's national observatories, DAO and DRAO , serves a number of purposes, including training students, contributing to science and supporting the development of instrumentation projects for Canada's international observatories. Examples of how Canada's national telescopes are contributing to new research and discoveries include:

  • The DAO 1.8m Plaskett Telescope was fitted with a new polarimeter module (dimaPol) in 2012which is being used to detect new magnetic stars and determine rotation periods and longitudinal magnetic field curves (Bohlender & Monin, 2013; Monin, Bohlender, Hardy, Saddlemyer, & Fletcher, 2012).
  • Observations using the DAO 1.2m telescope are automated and are fitted for remote observing mode, allowing users to access the telescope remotely through the Internet. This feature has increased the telescope's subscription rate by 50percent as telescope users no longer need to travel to the DAO to conduct their observations (Monin, Saddlemyer, & Bohlender, 2014).
  • The DRAO 26m John A. Galt Telescope is one of the best telescopes in the world for studying the role of magnetic fields in the structure and physics of the interstellar medium (the matter that exists between star systems).
  • DRAO 's automated Solar Radio Flux Monitor provides values of the 10.7cm Solar Flux solar activity index to the CSA and Natural Resources Canada ( NRC an), allowing data to be used by a number of agencies around the world, including the National Aeronautics and Space Administration ( NASA ), the National Oceanic and Atmospheric Administration, and the US Department of Defense ( NRC an, 2015).
  • The DRAO site is host to instruments from other partners requiring a radio-quiet environment, such as the university-led Canadian Hydrogen Intensity Mapping Experiment ( CHIME ) radio telescope, currently under construction, which will be used to measure the expansion of the universe.
  • The national observatories also play a role in public outreach.

The DAO and DRAO facility staff also support instrumentation and technology development for international observatories supported by Canada and for other observatories around the world. Instrument development and integration for international telescopes is one of HAA 's major activities, and one for which they are widely recognized. Close consultations between engineering and scientific staff from HAA programs during the instrumentation development process ensure the relevance and success of the instruments developed. For example, approximately 70percent of the research time available at the Gemini Observatory employs instrumentation designed and constructed by HAA staff. More information about HAA 's contribution to Canada's global standing in astronomical research, particularly as it relates to the development of instruments and technologies, can be found in Sections 4.1.1, 4.1.5 and 4.1.8.

3.1.3Alignment of current suite of international telescopes with the needs of Canadian astronomers

Key Finding 4: The current suite of telescopes supported by Canada is well aligned with the needs of Canadian astronomers.

Level of alignment

The evaluation found that Canada's participation in international telescope agreements is aligned with the needs of Canadian astronomers. In fact, nearly all survey respondents (89percent)"strongly agree" or"agree" that accessing international telescopes supported by Canada is critical to meeting their research needs. Survey respondents and key informants had a generally positive perception of telescopes supported by Canada. More than two thirds of survey respondents (68percent) reported that the current suite of telescopes supported by Canada is aligned with their research needs. In terms of optical telescopes, Canada's involvement in CFHT (4m class telescope) and Gemini (10m class telescope), as well as its future involvement in TMT (30m class telescope), were described by several key informants as critical for Canada's role in advancing astronomy. For radio astronomy, some external key informants also commended Canada for its involvement in ALMA , which was described as having the potential to create whole new fields of astronomy.

A few key informants, however, expressed dissatisfaction with the astronomy community's decision to withdraw from the JCMT , noting that Canada's smaller share in ALMA has led to challenges for Canadian astronomers applying for observing time. These same interviewees admitted the decision to withdraw from JCMT was widely accepted in the astronomy community and that NRC had adequately implemented the community's decision, as expressed in the LRP2000 and reiterated in the LRP2010 .

Subscription rates

The standard measure of community demand for access to an observatory is its subscription rate, i.e. , the ratio of time requested to time available (STF, 2014, p. 22). A subscription rate of greater than one (1.0) is an indication that demand exceeds time available.

Over the period of the evaluation, the average Canadian subscription rate for the international telescopes Canada supports was 3.03, well above HAA 's target subscription rate of 1.5. As shown in Figure1, Canada's offshore observatories were generally oversubscribed during this period and rarely fell below HAA 's target subscription rate.

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (2)

Long description of Figure1.

Figure1is a vertical bar chart comparing the subscription rates for CFHT , Gemini North and South and JCMT to HAA 's target subscription rate of 1.5.

You'll find increasing
SemesterSubscription rate
CFHT Gemini NorthGemini SouthJCMT Table note *
2011A2.262.522.031.51
2011B1.551.851.551.01
2012A1.692.221.405.59
2012B1.782.623.088.09
2013A1.582.971.522.57
2013B1.432.182.303.80
2014A1.252.562.402.12
2014B1.802.683.55-
2015A2.882.282.07-
2015B1.462.641.55-

Source: HAA performance data

Table notes

Table note *

Subscription rates for JCMT are not available after semester 2014A since NRC withdrew from the JCMT partnership in September 2014.

Return to table note * referrer

ALMA has also been consistently oversubscribed since early science observations began in 2011. ALMA 's time allocation process is based on cycles (which can vary from 9to 16months), which is much longer than a semester (usually 6months). Therefore, data on subscription rates for Canadian Principal Investigators ( PI s) to the observatory is shown separately in Table4. The Canadian subscription rate for ALMA is high, averaging approximately 5.7between Cycles 0to 3, well above HAA 's target.

Table4: Canadian PI subscription rate for ALMA by cycle
CycleStart dateSubscription rate
Cycle0September 201111.68
Cycle1January 20133.33
Cycle2June 20143.89
Cycle3October 20153.91

Source: HAA performance data.

While Canada's offshore observatories were generally oversubscribed between 2011and 2015, there have been notable fluctuations in the subscription rates over the past five years (as shown in Figure1 and in Table4). Aging instruments, the commissioning of new instrumentation, and moves by some observatories to larger survey programs, were identified by observatory partners as factors that may have an impact on subscription rates.

Access to other telescopes not supported by Canada

While evidence of demand for access and findings from key informant interviews reinforce the rationale for investing in Canada's current suite of telescopes, it should be noted that Canadian astronomers also access telescopes not supported by Canada. Results from the survey show that more than two thirds of respondents (69percent) were awarded time on ground-based telescopes not supported through HAA 's ITA Program between 2011and 2015. Overall, 62different observatories were accessed by survey respondents. When asked to describe the reasons for accessing these telescopes, approximately three quarters of respondents (76percent) reported that the telescopes have instruments or capabilities that are not available with the telescopes currently supported by Canada.

Two key factors influence the decision to access telescopes outside Canada's current suite of observatories:

  • a desire for multiple data sources from a diversity of telescopes and instruments
  • the need to access telescopes with unique capabilities or which provide access to specific parts of the electromagnetic spectrum

Canadian astronomers gain access to other facilities either through the"open time" allocation (or"open skies" policy) of other observatories or through another country's allocation (by joining an international team). The concept of open access is fundamentally based on quid pro quo relationships between collaborators, whereby countries will gain access to other facilities as long as they have something to offer in return. Therefore, as long as Canada is perceived as"pulling its own weight" in terms of its support for science ( i.e. , by providing financial allocations to observatories through ITA or by developing telescope instrumentation) then Canadian astronomers will continue to be welcomed collaborators.

3.1.4Astronomy community's needs in relation to accessing telescope data

Key Finding 5: In order for Canadian astronomers to conduct ground-breaking scientific research in astronomy, there is continued need for Canada to invest in research data management infrastructure and capabilities ( e.g. , data storage, cataloging, networking, high-performance computing, software and data management skills).

Need for storage and the provision of access to data

"You'll find increasing generations of young astronomers who want access to that data. It will be important years from now […] I just want to emphasize that this is a key component. We want to be sure that the whole entity is there for later generations. This is how discoveries are made."

The Canadian Astronomy Data Centre ( CADC ) is the access point for researchers to retrieve their data after observations have been executed on many of the supported telescopes. However, ALMA and Gemini currently use their own data storage systems. In the case of specific observatories, such as CFHT , JCMT and other international telescopes, the CADC is responsible for protecting NRC investments by providing security for the data they produce. When considering the costs to plan, build and operate the supported telescopes, the CADC archives contain hundreds of millions of dollars' worth of data gathered over the last 40years.

As shown in Figure2, there has been an exponential increase in the data delivered to users over the last 10years. During the period covered by the evaluation, the amount of data delivered annually by the CADC increased 79percent.

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (3)

Source: HAA performance data
Note: Data used does not distinguish between Canadian and global use of the CADC .

Long description of Figure2

Figure2is a vertical bar chart which shows the increase in the data delivered by the CADC (in millions of gigabytes) to users between 2005and 2015. The figure shows that the amount of data delivered by CADC users has increased significantly during this period.

YearGigabytes delivered
200538,950
200649,783
200752,326
2008116,997
2009114,997
2010168,124
2011923,258
20121,063,855
2013920,037
2014740,801
20151,649,251

Figure3 illustrates the"Data to Knowledge" Flow Diagram, a simplified visualization of how HAA supports the astronomy community through the CADC .

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (4)

Long description of Figure3

Figure3illustrates the "Data to Knowledge" Flow Diagram, a simplified visualization of how HAA supports the astronomy community through the CADC . The CADC provides support to the astronomy community in four areas:

  • Data collection
  • Data storage (Area of CADC support)
  • Data cataloguing (Area of CADC support)
  • Data distribution (Area of CADC support)
  • Data processing (Area of CADC support)
  • Science reporting
Data Collection:
Observatories collect data under optimal weather conditions.
Data Storage:
Raw data is stored and data integrity is maintained by CADC .
Data Cataloguing:
Data is sorted and catalogued by CADC staff so that astronomers can easily search through and access the data that they need.
Data Distribution:
Data is downloaded by astronomers. Due to the large volume and traffic of data, there is a need for high speed networking to meet demand.
Data Processing:
Raw data is processed into meaningful information for use in generating new knowledge about the universeFootnote 4. For some astronomers high-performance computing ( HPC ) is a critical component for processing this data. It should be noted that for certain observational data, CADC pre-processes the data prior to distribution so that it is science-ready, these are referred to as"advanced data products".
Science reporting:
The last step refers to the publication of papers.

" CADC is fabulous. It allows me to do science rather than get caught up in the complexities and intricacies of the data. One can access data from other places, but using the CADC as the conduit has been the most satisfactory."

Internal and external key informants noted that there is a need for"advanced data products" (science-ready data). Processing larger sets of data ("big data") requires advanced knowledge in data management, such as the development of novel algorithms, which may be beyond the expected skill levels/areas of expertise of a given astronomer. For example, the Canadian astronomy community has a strong scientific interest in conducting large sky surveys that generate vast amounts of data. This is a driving force behind the need for pre-processed"advanced data products".

Need for advanced data management infrastructure

Interviewees indicated that Canada does not currently have the needed data management infrastructure ( i.e. , networking, data processing and storage) in place to manage data for the next generation of world-class observatories ( e.g. , SKA ). The needs of the astronomy community have shifted beyond simple data storage and distribution to high-speed networking, high-performance computing, and the skills and software needed to process and analyze large volumes of data. Evidence from the evaluation indicates that the scale of observational data necessary to conduct leading-edge scientific research in astronomy is growing. As observatories become increasingly complex there is an accompanying need for complex infrastructure, such as data management equipment and services, networks, software and skills, to distribute and process large and complex observational datasets. This was the impetus for Canadian Advanced Network for Astronomical Research ( CANFAR ), an HPC cloud computing network solution for advanced research in astronomy ( CASCA , 2016, p. 96). By working with partners to support CANFAR , HAA helps enable remote access to high-performance computing, throughput, and data storage for advanced data products.

The management and funding of Canada's high-performance computation and networks is a complex issue that spans local, provincial and federal jurisdictions, and implicates other research disciplines with similar needs for advanced computing and networking ( CASCA , 2010, p. 49). Documents reviewed from CASCA 's Computation and Data committee argued that"(f)uture research goals in astronomy will be held back without a consistent approach, and long-term, reliable funding for computing, data, and networks" ( CASCA , 2012, p. 22). For example, the SKA telescope will place exceptional demands on processing and storage, producing petabyte to exabyte datasets, with daily data collection expected to be approximately 10times the data of the current global internet traffic ( SKAO , 2016). This demonstrates the continued need for advanced data management infrastructure to support ongoing access to astronomy data. Additional details regarding the CADC 's current capabilities and challenges related to infrastructure and networking capacity are discussed in Section 4.2.4.

3.2Appropriateness of current delivery model

To assess the relevance of HAA , the evaluation asked key informants whether other potential delivery models exist that could better service the needs of Canadian astronomers.

Key Finding 6: There are benefits associated with the current delivery model. No other delivery model was identified as more appropriate for the administration and operation of Canada's telescopes.

Key informants agreed that the current delivery model, whereby HAA is responsible for the administration and operations of Canada's telescopes, is appropriate. They also commended HAA 's"excellent" work in supporting the Canadian community over the last five years.

These findings are contrary to the MTR 2016which indicated that"serious consideration should also be given to the possibility that astronomy in Canada might be better served by an entirely new national organization" ( CASCA , 2016, p. 98). According to the MTR 2016, this new organization could be created to address two key issues identified by the astronomy community in the report ( CASCA , 2016, p. 99):

  • the lack of clear and formal processes for the community to lobby the Canadian government for future telescope opportunities.
  • the perceived impact of NRC 's increased focus on industrial innovation on HAA 's ability to deliver its mandate.

When asked whether a new model would address these issues, key informants did not agree with the MTR 2016and noted that changing the model would not necessarily be a panacea for these problems. In terms of alternatives, a few key informants mentioned that ACURA could, hypothetically, play a role if the delivery model was to be reviewed. However, one interviewee noted that under such a scenario, ACURA would"need a significant evolution" and additional funding would be needed to pay for the increase in its operations.

While a more appropriate alternative model to the administration and operation of Canada's telescopes was not identified by key informants, the evaluation found that there are advantages associated with the current model:

  • Financial flexibility: Being part of NRC allows HAA to access other internal sources of funding to minimize the financial impact resulting from devaluation of the Canadian dollar against other currencies.
  • Ability to negotiate and engage in international agreements: As a federal government organization, NRC has the legislative authority to sign international agreements with third party organizations and to administer Grant and Contribution (G&C) funding.
  • Neutrality: As a national entity, HAA equally represents the interests of all Canadian universities involved in astronomy, regardless of their size.

Some key informants noted that having a federal laboratory fully dedicated to operating facilities and conducting astronomy and instrumentation research is an excellent model. They explained that as the observatories and their instrumentation increase in value, duration and complexity, the stability and core expertise provided by a federal lab are particularly important.

3.3Alignment with federal priorities and NRC strategic objectives

The evaluation assessed the extent to which the objectives and activities of HAA were aligned with federal government priorities and NRC 's strategic objectives. It also examined the effect of NRC 's transformation on HAA 's outreach activities.

3.3.1Alignment with federal priorities

Key Finding 7: HAA 's activities, including investment in international observatories, are aligned with federal government priorities.

While astronomy was not explicitly identified as a key priority for the previous government, the announcement of the investment in the TMT project in Federal Budget 2015underlined the importance of investing in astronomy projects. Former Prime Minister Stephen Harper announced that Canada was proud to be an official partner in the TMT and that the project"will advance Canadian and international scientific discovery" as well as "generate new capabilities and technologies in Canada which will help create and maintain high-quality jobs in communities across the country" (Government of Canada, 2015).

While the new strategic priorities of the current government have yet to be fully articulated in a formal strategy, the Minister of Science's Mandate Letter from the Prime Minister requests that the new Minister:"examine options to strengthen the recognition of, and support for, fundamental research to support new discoveries" (Office of the Prime Minister, 2015).

Further, the CADC initiative is aligned with Canada's federal science priorities, as its operations are innovative in the field of"big data'" and promote open science. The 2014Federal Science and Technology Strategy stated that"nations must swiftly … harness large and complex data systems ("big data") and adopt"open science" policies to foster collaboration" (Industry Canada, 2014, p. 1).

3.3.2Alignment with NRC 's strategic objectives

Key Finding 8: Although HAA 's mandate is reflected in NRC 's Strategy, the alignment between HAA 's activities and the objectives of NRC and the Emerging Technologies Division is unclear.

Alignment with objectives of NRC and the Emerging Technologies Division

NRC 's Strategy 2013-2018focuses on market-driven innovation outcomes and benefits to Canada. The strategy describes four business lines, one of which includes the provision of access to world-class science infrastructure, enabling clients to effectively use some of Canada's most specialized, large-scale scientific infrastructure. This aligns with HAA 's mandate. However, according to many internal key informants, HAA is not aligned with NRC 's overall strategic goal to "spur economic prosperity by delivering technology development programs and specialized national facilities and services, all with an eye to boosting industrial R&D ". Further, some internal key informants feel that HAA 's core mandate, to provide access to scientific infrastructure to generate key discoveries and scientific knowledge, is not aligned with the objective of the Emerging Technology Division, which is "to understand, anticipate and build capacity to address emerging markets with technologies essential to Canada's future industrial and societal needs".

Evidence shows that HAA is supporting industrial R&D through the engagement of Canadian firms to develop, manufacture and install telescope instrumentsV for national and international observatories. However, its primary aim is to serve the astronomy community.

3.3.3Impact of NRC 's transformation on HAA 's outreach activities

Key Finding 9: HAA 's ability to conduct outreach activities was limited over the last five years, particularly as a result of the decision to close the Centre of the Universe.

Public outreach and awareness are considered by HAA stakeholders as an integral part of the Portfolio's activities. NRC 's transformation to a Research and Technology Organization ( RTO ) in 2012primarily affected HAA by changing its organizational structure (from an institute model to a portfolio model). According to internal stakeholders, this change has limited its ability to engage in outreach activities.

Additionally, the evaluation found that some NRC -level decisions largely affected HAA 's outreach activities, particularly the 2013 NRC decision to close the Centre of the UniverseFootnote 5. This decision was based on financial considerations and the view that the public outreach mission of the Centre of the Universe was not aligned with the mandate of the HAA programs. The Portfolio is exploring alternative operating models for the Centre of the Universe, such as, for example, by recently signing a License to Occupy with the Friends of the Dominion Astrophysical Observatory, a non-profit organization, for use of the Centre of the Universe for public outreach purposes. However, in spite of these efforts, access to the centre continues to be limited.

Recommendation 2: HAA should continue to work with partners to expand outreach and awareness activities.

3.3.4Collaborations with other NRC Portfolios

Key Finding 10: There were some informal interactions between HAA and other Portfolios over the last five years. While future opportunities for collaborations and knowledge exchange may exist, HAA and the other Portfolios of NRC need to better understand each other's capabilities, activities and needs.

There were no formal collaborations between HAA and the other NRC Portfolios from 2011to 2015. As one indicator of formal collaboration, SAP data provides information on labour sharing between portfolios ( i.e. , labour in dollars received from and sent to other portfolios) for FY s 2013-14and 2015-16, the only period for which data is available. Overall, the data shows that the average amount of labour shared between NRC portfolios over the three-year period was approximately one million dollars per portfolio per year, whereas HAA was shown to have none. According to internal key informants, this dissimilarity can be explained by the high rate of staff utilization, a lack of awareness of competencies available in other portfolios and difference in mandates. However, key informants provided some anecdotal evidence of informal interactions with other NRC portfolios. For example, a key informant noted that HAA provided low noise amplifiers ( LNA s) to support research by the Security and Disruptive Technologies (SDTech) Portfolio.

Further, one key informant indicated that attending the NRC 's Tech-X event helped increase his awareness of what was available and potential opportunities for collaboration in the field of photonics. However, he believed that his team's current workload would not allow him to pursue these potential collaborations.

Recommendation 3: HAA , in collaboration with other NRC Portfolios and NRC Business Management Support ( BMS ), should explore opportunities to leverage competencies and astronomy-related technologies.

3.4Alignment with federal roles and responsibilities

The evaluation assessed the extent to which the strategic objectives and activities of HAA were aligned with federal government roles and responsibilities.

Key Finding 11: HAA 's activities and strategic outcomes are aligned with federal roles and responsibilities, as expressed in the National Research Council Act.

"Without an organization like [ HAA ], Canada could not be a credible and productive partner in major international observatory facilities."

The National Research Council Act states that the Council may"operate and administer any astronomical observatories established or maintained by the Government of Canada" (Justice Canada, 2009). Documentation reviewed as part of the evaluation support the role played by the federal government in administering Canada's national and international telescopes. Large-scale, multi-national endeavours generally require the federal government to be a signatory and NRC has a well-recognized role as the Canadian signatory in international astronomy agreements.

4.0Performance

The evaluation assessed HAA 's performance in terms of its effectiveness (Section 4.1) and efficiency and economy (Section 4.2).

4.1Effectiveness

To assess HAA 's effectiveness, the evaluation examined the Portfolio's success in providing Canadian astronomers with access to national and international observatories and science data products, its involvement in next-generation telescopes, its contribution to the development of observatory instrumentation, the industry impacts of instrumentation projects, as well as the productivity and scientific impact of Canada's offshore telescopes.

4.1.1Access by Canadian astronomers to national and international telescopes

Observation time awarded to Canadian astronomers

Key Finding 12: HAA 's involvement in offshore observatories, mainly through financial support and contributing to the development of instrumentation and infrastructure, is critical to ensuring that Canadian astronomers are able to access international telescopes to conduct their research.

Key Finding 13: Observing time awarded to Canadian astronomers at ALMA is not commensurate to Canada's financial contribution to the observatory's operations.

CFHT , Gemini and JCMT

Through its funding support to observatory operations, Canada is guaranteed a proportion of the total time available for each facility's telescope. Currently, Canada is allocated 42.5percent of the observing time available at CFHT and 17.8percent of the observing time available at Gemini ( CFHT , n.d. ; Gemini Observatory, 2014). Before withdrawing from the JCMT partnership in 2014, Canada's share of the observing time at JCMT was 25percent ( CASCA , 2014a). HAA representatives confirmed that the time awarded to Canadian astronomers at CFHT , JCMT and Gemini during the period covered by the evaluation was consistent with the fraction of time earmarked for Canada at the observatories.

ALMA

ALMA is a partnership of Europe, North America and East Asia in cooperation with the Republic of Chile. The North American portion of the partnership, which includes NRC HAA , the NRAO and the National Science Council of Taiwan, provides annual funding to ALMA representing approximately 37.5percent of the total funding allocation from all ALMA partners. Canada's annual funding is approximately 7.25percent of the North American portion of ALMA funding. Canada's allocation gives Canadian astronomers access to the North American portion of ALMA time. Unlike CFHT and Gemini, there is no portion of time earmarked specifically for Canadian astronomers.

As discussed in Section 3.1.3, the Canadian PI subscription rate for ALMA time is significant, averaging approximately 5.27between Cycles 0to 3, which ran from 2011to 2015. During this period, Canadian PI s were awarded a total of 66.2observing hours at the facility, representing approximately 4percent of the hours awarded to all North American researchers during this period. However, data indicates that the proportion of time awarded to Canadian PI s is not commensurate to Canada's annual funding support to ALMA . Data is presented in Figure4.

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (5)

Source: HAA performance data.

Long description of Figure4

Figure4is a vertical bar chart which compares the observing time awarded to Canadian astronomers and Canada's funding support to ALMA . The figure shows that the time awarded to Canadian PI s, as percentage of hours awarded to North American astronomers, has decreased during the evaluation period.

CycleNumber of hours awarded to Canadian PI sHours awarded to Canadian PI s as a proportion of hours awarded to North America
Cycle016.910.2%
Cycle111.24.2%
Cycle220.43.2%
Cycle317.72.6%

It should be noted that ALMA time allocated to PI s is based on merit and through a peer review process that includes Canadian astronomers as proposal reviewers. It should also be noted that Canadian astronomers have been successful in obtaining observing time at ALMA through other means, mainly as collaborators, or co-Investigators, on other research teams, including those outside North America. Overall, Canadian co-Investigators were awarded a total of 581hours between 2011and 2015, which represents 11.3percent of the global time allocation at ALMA during this period. Additionally, Canadian astronomers were co-authors on 13of the top 30most cited ALMA papers.

While the time allocated to Canadian PI s has decreased during the evaluation period, HAA management indicated that ALMA is a newly operating telescope (operational since 2011) and that the allocation to Canadian PI s is expected to improve with time. However, since there is no portion of time earmarked specifically for Canadian astronomers ( i.e. , commensurate with Canada's funding support to ALMA ), further consideration should be given to identifying barriers, if any, that are impacting Canadian PI access to the observatory and finding potential solutions to improve access.

Recommendation 4: HAA should identify the reasons why Canadian PI s are not allocated observing time commensurate with Canada's financial support to ALMA operations and take the appropriate measures identified, as needed.

HAA 's involvement in and contribution to next generation telescopes

Key Finding 14: HAA 's participation in the pre-construction and construction phases of next generation telescopes is important to securing Canadian access to observatory facilities.

The extent to which a country's researchers are able to gain access to an observatory facility is related to that country's financial and in-kind contributions to the facility. HAA and the Canadian astronomy community are significantly involved in the planning of two major new observatories, TMT and SKA . A brief description of Canada's involvement and contributions to both observatories is provided below.

TMT

The TMT project began in 2003as a collaboration among ACURA , the California Institute of Technology (Caltech), and the University of California (UC). Federal and provincial partners, including NRC , provided an estimated $31million in funding to support the design and pre-construction phases of the project. These pre-construction activities were managed by ACURA in consultation with HAA . In April 2015, the Government of Canada announced an allocation of $243.2million over 9years to NRC for the construction and commissioning of TMT and, consequently, NRC assumed responsibility for Canada's participation in the observatory. Further, representatives from the Canadian astronomy community and HAA are members of the TMT 's Board of Governors and Science Advisory Committee.

Through its participation, Canada is making two major contributions to the construction of the TMT :

  • TMT enclosure: Dynamic Structures Ltd., located in Port Coquitlam, BC, is responsible for the design and construction of the enclosure. The observatory enclosure will be spherical with a circular aperture to match the telescope reflector's field of view, which will be essential in preserving the TMT 's image quality and will reduce the mass of the facility, resulting in lower costs ( CASCA , 2014c, p. 14).
  • Narrow Field Infrared Adaptive Optics System ( NFIRAOS ): HAA , in collaboration with industry, is developing the AO system for TMT . The system is integral to the TMT observatory as it corrects degradations in image quality that result from observing objects through the Earth's atmosphere and from aero-thermal and optical imperfections within the observatory itself, resulting in sharp and stable images ( CASCA , 2014c, p. 7). Further, NFIRAOS will support other TMT instruments, ensuring that the instruments achieve their specified performance ( CASCA , 2014c, p. 9).

Portfolio documentation also indicates that an Integration and Test (I&T) facility is expected to be built in Victoria to support TMT instrumentation projects. These contributions will provide Canadian astronomers with access to the telescope and will enable Canada to play a significant role in defining the future operation and capabilities of TMT ( CASCA , 2014c, p. 14).

Construction of the TMT began in October 2014on Mauna Kea in Hawaii. However, on-site work has been on hold as of December 2015Footnote 6.

SKA

Canada, represented by NRC , formally joined the SKA Organization ( SKAO ) in early 2012and is committed to delivering in-kind work for the SKA pre-construction phase, which began in 2013. The ACURA Advisory Council on the SKA (AACS), which includes representatives from universities and industry, is responsible for overseeing Canadian participation in SKA .

As part of Canada's contribution to the pre-construction design work for SKA , HAA is contributing to the following projects ( CASCA , 2014b, pp. 3–4):

  • Digital signal processing: HAA is leading the Central Signal Processor ( CSP ) Consortium for designing the correlators and beamformers as well as non-imaging pulsar search and timing processors.
  • Phased-array feeds ( PAF ): HAA is involved in the work to demonstrate PAF s on dishes, focusing on cryo-cooled PAF s.
  • Low noise amplifiers ( LNA ) and RF digitizers: HAA has developed cryogenic LNA s that will be part of the EMSS L-band and Ultra high frequency ( UHF )-band receivers on MeerKAT. Further, as part of the Dish Consortium work, HAA is responsible for the digitizers for single-pixel feed.
  • Telescope management: HAA is involved in the design of the SKA 1monitor and control system as well as interfaces of the SKA Telescope Manager.

Additionally, the Canadian astronomy community and HAA have representation on SKA 's Board of Directors, Science and Engineering Advisory Committee, Science Review Panel and 11of the 13 SKA science working groups.

4.1.2Access by Canadian astronomers to science data products

Key Finding 15: Through the CADC , HAA has been successful in providing astronomers with access to science data. Further, CADC 's data management has contributed to increasing research productivity (scientific output) of Canada's offshore telescopes.

Data management services provided by the CADC

HAA uses the 'number of refereed publication acknowledgments of the CADC ' as a proxy indicator for its contribution to scientific impact in astronomy. The annual number of acknowledgments referring to the CADC has increased by 55percent over the last five years, going from 120in 2011to 186in 2015. The increase in data delivered to users (as shown in Figure2 in Section 3.1.4) and global acknowledgments of the CADC in astronomy literature demonstrate that the CADC is successfully providing astronomers with access to science data and advanced data products.

The evaluation identified several notable scientific impacts resulting from the use of CADC archives, such as:

"[The CADC ] is more than just a data store; they are a partner in how we get data out, and publications out."

  • Archival research using CFHT data: Between 2013and 2015, research papers utilizing CFHT archival data represented over a third of CFHT papers produced ( CFHT , 2016c). These publications show that the CFHT — despite being a 4m class telescope operating in an environment of 8-10m class telescopes — remains scientifically competitive in terms of publications output (as shown in Figure5 in Section 4.1.3).
  • Identification of instrument degradation at CFHT : In late 2014, a CADC staff member used archival data to identify a discernible drop in the amount of light collected by a key instrument, leading to the early discovery and subsequent repair of a degrading component in its most popular instrument ( CFHT , 2016a; CADC , 2016b).
  • The New Horizons space probe: Launched by NASA in early 2006, the probe visited the dwarf-planet Pluto in July 2015. Archival data from the CADC fed into NASA 's navigation process, aiding the guidance of the probe on a collision-free approach during its encounter ( CFHT , 2015; CADC , 2016a).
Development of advanced data products

The evaluation highlighted several HAA contributions to the development of advanced data products. Key informants described the CADC 's work in providing advanced data products as follows:

  • Data provided by the CADC is "science-ready", making it more useful for researchers.
  • CADC is one of the few astronomy data archives that provides semi-processed stacked data, describing it as "a mine ready to be tapped."
  • Large surveys, such as the CFHT Legacy Survey, are critically dependent on the data management services provided by the CADC to process the data.
  • CADC 's search and imaging preview capabilities are useful as they enable astronomers to efficiently judge if the quality of underlying data is sufficient to answer a specific scientific question.

CANFAR 's cloud computing solution is a key evolution in how the CADC serves the Canadian astronomy community. Since 2011, over 500publications cite CANFAR support ( CASCA , 2016, p. 96). The evaluation identified several notable results from HAA 's collaboration in CANFAR , such as:

  • The Next Generation Virgo Cluster ( NGVC ) survey gathered a large data set from over six years of observations with the CFHT . The CADC , through CANFAR , provided support to the NGVC team enabling them to archive and process their massive data set (University of Victoria, 2012).
  • In 2014, CADC data analyzed through CANFAR led to the identification of a Kuiper Belt Object ( KBO ) with a diameter of approximately 30km and within reach of NASA 's New Horizons space probe. NASA may visit this object in January 2019(CANARIE, 2014).
Factors hindering HAA 's ability to provide access to science data products

The evaluation identified two main factors hindering HAA 's ability to provide the astronomy community with access to science data products:

  • outdated IT infrastructure, such as servers and storage systems equipment, supporting the CADC .
  • limited network capabilities available to the CADC (there is a need to upgrade from one gigabit per second to 10gigabits per second).

Additional details regarding these issues are provided in Section 4.2.4.

4.1.3Observatory productivity and impact

Key Finding 16: Canada's offshore observatories have performed well internationally in terms of productivity and impact ( i.e. , the relevance of observatory publications).

As one indicator of performance, the evaluation consulted various bibliometric data provided by HAA to assess the performance of Canada's offshore observatories, specifically in terms of publication outputs and impacts. The evaluation's findings are provided below.

Productivity– number of papers produced annually

The number of papers produced annually by a telescope represents a measure of its contribution to the generation of new knowledge and the return on capital investment for the construction of the telescopes and their instruments (Crabtree, 2008). To provide an accurate assessment of an observatory's performance, productivity is examined in conjunction with the observatory's impact, which is measured through citation counts of the papers it produces. Observatory impact is discussed in greater detail in the next section.

Figure5 presents the productivity, in terms of papers produced annually, of Canada's offshore observatories and 21other international telescopes between 2010and 2014. Overall, two of Canada's observatories were ranked amongst the top-10in terms of productivity during this period: CFHT ranked fifth and Gemini ranked ninth. While ALMA is twenty-first overall, the observatory's productivity has increased progressively year over year since it began operations in 2011and is expected to continue to do so as the observatory's scientific operations ramp up.

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (6)

Long description of Figure5

Figure5is a vertical bar chart which compares the productivity ( i.e. , the number of papers produced annually based on data from the observatory) of Canada's offshore observatories ( CFHT , Gemini, JCMT and ALMA ) and 21other international telescopes between 2010and 2014. Some observatories have more than one telescope. In order to normalize the results, the number of papers produced by each observatory was divided by the number of telescopes. For example, VLT consists of four telescopes, while Gemini, Magellan and Keck have two.

ObservatoryNormalized number of papers produced
VLA/ JVLA 967
UKIRT 829
KeckTable note *767
VLT Table note *741
CFHT 660
Subaru631
NTT 573
IRTF 524
GeminiTable note *457
Mayall456
JCMT 445
MMT 429
MagellanTable note *384
Blanco351
GBT 345
AAT 341
ESO3p6 332
TNG 307
Arecibo189
WIYN184
ALMA 182
SOAR 156
HET 148
LBT 137
SALT 83

Source: HAA performance data.

Table notes

Table note *

Some observatories have more than one telescope. In order to normalize the results, the number of papers produced by each observatory was divided by the number of telescopes. For example, VLT consists of four telescopes, while Gemini, Magellan and Keck have two.

Return to table note * referrer

Impact– count of citations

Citation counts ( i.e. , the number of times a paper is cited in literature) are used to measure an observatory's impact in terms of the relevance of the papers it produces (Crabtree & Zhang, n.d. ). Figure6 presents the medianFootnote 7 number of citations per paper for Canada's offshore observatories and 21other international telescopes. Overall, data indicates that three of Canada's observatories are amongst the top-10in terms of median impact per paper, with ALMA ranked first overall followed by CFHT in seventh place and Gemini in tenth place.

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (7)

Long description of Figure6

Figure6is a vertical bar chart which compares the median impact per paper ( i.e. , median number of citations per paper) of Canada's offshore observatories ( CFHT , Gemini, JCMT and ALMA ) and 21other international telescopes between 2010and 2014.

ObservatoryMedian impact per paper
ALMA 3.29
Keck3.1
HET 2.75
Magellan2.6
MMT 2.33
Mayall2.3
CFHT 2.28
UKIRT 2.25
LBT 2.2
Gemini2.17
ESO3p6 2.17
WIYN2
SOAR 1.85
GBT 1.85
Blanco1.83
AAT 1.83
VLT 1.8
Subaru1.75
TNG 1.75
NTT 1.75
IRTF 1.75
JCMT 1.75
VLA/ JVLA 1.62
Arecibo1.58
SALT 1.46

Source: HAA performance data.

Another measure of relevance is the number of "high-impact papers" produced by an observatory ( i.e. , papers in the 90th percentile for number of citations or the top 10% most cited papers).Figure7 presents the percentage of papers that are considered "high-impact" for Canada's offshore telescopes and other international observatories. Overall, two of Canada's observatories are amongst the top-10observatories, including ALMA in first place and CFHT in sixth place.

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (8)

Source: HAA performance data.

The Long description of Figure7

Figure7is a vertical bar chart which compares the percentage of high-impact papers ( i.e. , papers in the 90th percentile for number of citations or the top 10% most cited papers) of Canada's offshore observatories ( CFHT , Gemini, JCMT and ALMA ) and 21other international telescopes between 2010and 2014.

ObservatoryPercentage of high-impact papers
ALMA 17%
Keck16%
Magellan13%
MMT 12%
GBT 12%
CFHT 12%
Arecibo12%
UKIRT 11%
Subaru11%
LBT 10%
HET 10%
Mayall10%
Blanco9%
WIYN9%
Gemini9%
ESO3p6 9%
SOAR 8%
IRTF 8%
VLT 8%
VLA/ JVLA 7%
TNG 7%
AAT 7%
JCMT 6%
NTT 6%
SALT 2%

4.1.4Examples of scientific impact and key discoveries

Key Finding 17: The use of Canada's international observatories has resulted in a number of important discoveries over the last five years.

Evidence indicates that the use of Canada's offshore observatories has resulted in significant discoveries over the last five years. Data gathered from program documentation and key informant interviews provide a number of examples of scientific impact resulting from the use of Canada's international telescopes. Examples of key discoveries are presented in Table5.

Table5: Examples of discoveries resulting from the use of Canada's offshore observatories
DiscoveryDescription
ALMA
Images of protoplanetary disc surrounding HL TauriIn 2014, astronomers captured, what is considered to be, the best ever image of planet formation around an infant star. The image revealed, in fine detail, the planet-forming disc surrounding HL Tauri, showing multiple concentric rings separated by clearly defined gaps. These observations provide valuable information on the planet-formation process ( ESO , 2014; NRAO , 2014).
CFHT
Discovery of dwarf planet 2015RR245In February 2016, an international team of astronomers, led by a researcher from HAA , discovered a dwarf planet, approximately 700km in diameter, orbiting in the Kuiper Belt. 2015RR245is one of the largest worlds beyond Neptune to be identified. This discovery, coming from the Outer Solar System Origins Survey ( OSSOS ) which uses CFHT 's MegaPrime/MegaCam, allows astronomers to gain a better understanding of the history of the Solar System ( CFHT , 2016b).
Pan-Andromeda Archaeological Survey ( PAndAS )The Pan-Andromeda Archaeological Survey ( PAndAS ), led by an HAA researcher and conducted at CFHT , provided the largest and deepest panoramic image ever obtained of a galaxy that is similar to the Milky Way. The project has resulted in the discovery of a large number of new structures:
  • Approximately 20new"satellite" galaxies that appear to map out an enormous"ring", and appear to be rotating uniformly around Andromeda. This coordinated behavior was completely unexpected and cannot currently be explained. It implies a fundamental misunderstanding of the process of how galaxies form.
  • Approximately 100new"globular clusters", each of which contains millions of stars. (The survey results increased the number that was known to exist by about 25%)
  • Many new stellar"streams" ( i.e. , the remnants of small galaxies that have since been"ripped apart" by Andromeda). While researchers suspected that this process was occurring, it was not clear how often it takes place; the survey results have proven useful in trying to understand the importance of this process.
Gemini
Discovery of supermassive black holes at the centre of galaxies NGC 3842and NGC 4889In 2011, observations using the Gemini North telescope provided evidence of the largest black holes ever measured in our nearby cosmological neighbourhood in the galaxies NGC 3842and NGC 4889. Both black holes are approximately 10billion times more massive than the Sun. The research is expected to provide information on how black holes and galaxies form over the history of the universe (Gemini Observatory, 2011; NSF , 2011).

For more comprehensive lists of discoveries consult the websites: ALMA , CFHT , Gemini

4.1.5Contribution to the development of novel instruments and technologies

Key Finding 18: HAA has been involved in and has made significant contributions to the design and development of world-class instruments and technologies for Canada's observatories and for other observatories around the world. Further, the utilization of these instruments has resulted in a number of scientific impacts and discoveries.

ATP staff have developed or contributed to the development of a total of 11instrumentation projects undertaken for seven observatories over the last five years. Three of the instruments developed by HAA since 2011were examined as part of the case studies, including:

HAA staff "have leading international reputations for developing innovative cutting-edge instrument designs and delivering highly reliable instruments on time and on-budget". As well, the instruments developed by HAA "are on a scale that cannot be duplicated in university labs."

  • ALMA Band 3cryogenic receivers: HAA collaborated with Canadian industry to develop and manufacture a suite of ultra-low noise 84–116 GHz cryogenic heterodyne receivers. The receivers play a role in array commissioning and are the most commonly used band for science observations. HAA is responsible for the ongoing maintenance of the receivers.
  • Gemini Planet Imager ( GPI ): An imaging instrument installed at the Gemini South telescope, the GPI achieves high contrast at small angular separations, allowing for the direct imaging and integral field spectroscopy of extrasolar planets around nearby stars. The instrument is considered the most advanced of its kind for imaging and analyzing planets around other stars. HAA staff provided the optical-mechanical structure, mechanical control software and system engineering for the instrument.
  • Wideband Interferometric Digital Architecture ( WIDAR ) Correlator: HAA staff were responsible for developing WIDAR , the signal processing correlator responsible for combining the signals from each of the Jansky Very Large Array's ( JVLA ) 27antennas. The correlator is one of the most powerful machines of its kind in the world. For example, it can perform 1016calculations per second (10petaflops) and has the capacity to process the equivalent of about 50million simultaneous telephone calls.

Findings from the case studies indicate that these instruments have greatly improved their respective observatories capabilities and have made significant contributions to a number of discoveries since 2011.

Some key informants characterized HAA as a world leader in designing and developing observatory instrumentation, particularly in adaptive optics systems, digital systems design and fine guidance sensors. For example, HAA 's design and development of the WIDAR correlator for the JVLA established its reputation in digital system design and delivery, and provided the experience which underpinned the Portfolio's involvement in the SKA CSP consortium ( CASCA , 2014b, p. 3).

4.1.6Industry collaborations and impacts

Key Finding 19: HAA 's involvement in international observatories has allowed Canadian companies to contribute to the development of telescope instrumentation and to benefit from financial returns and enhanced expertise and technological capabilities.

HAA engages industry primarily through the provision of procurement opportunities. Therefore, assessing HAA 's contribution to industrial R&D differs from other portfolios in that revenue may not be the most appropriate indicator of the Portfolio's engagement with and impact on industry.

Canada's participation in international telescope agreements provides Canadian firms with the opportunity to bid on Requests for Proposal issued by either the partner observatories or by HAA . Between FY 2011-12and FY 2015-16, HAA awarded a total of 177contracts, valued at approximately $10.4million, to 104institutions and companies for the provision of services to support various research and instrumentation projects. There was a peak in the number of contracts in FY 2012-13, mainly as a result of the addition of 36contracts for SKA -related projects, including the DVA1and CSP projects, valued at $1.6million. Contract data is presented in Figure8.

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (9)

Source: HAA administrative data.

Long description of Figure8

Figure8is a combined vertical bar and line chart which shows the number of value of contracts with industry and universities.

YearNumber of contractsValue
2011-1213$1.09M
2012-1371$3.89M
2013-1445$1.83M
2014-1523$2.37M
2015-1625$1.26M

Between FY 2011-12and FY 2015-16, nearly all of the contracts (88percent), valued at $9.04million, were awarded to private sector firms. Universities also benefited from contracts with HAA , with 22contracts, valued at roughly $1.4million, awarded to these institutions.

In addition to the financial benefits discussed above, evidence from the key informant interviews and case studies shows that HAA 's work with companies has strengthened their industrial research capacity and provided them with access to specialized equipment and training. Table6 provides examples of companies and organizations that have benefited from their involvement in telescope projects. Industry stakeholder interviewees noted that they would not have obtained the contracts for the instrumentation projects had it not been for HAA 's direct involvement in the observatories. They noted that it is difficult to obtain contracts for astronomy instrumentation and infrastructure projects outside of Canada as foreign governments and astronomy organizations tend to award contracts to domestic companies.

Table6: Examples of Canadian companies involved in telescope instrumentation projects
Name of FirmDescription of contribution and impacts
Nanowave Technologies Inc., Etobicoke, ONNanowave Technologies successfully produced highly customized components for the ALMA Band 3receivers (briefly described in Section 4.1.5). Nanowave representatives reported that, as a result of the company's involvement in the development of the ALMA Band 3receivers, they gained new capabilities, know-how and specialized skills related to the development of cryogenically cooled LNA s and precision wire bonding. NRC licensed the LNA technology to Nanowave, enabling the company to manufacture the amplifiers. Currently, Nanowave is working with HAA for the development of LNA s for MeerKAT radio telescope.
FiberTech Optica Inc., Kitchener, ONFiberTech Optica Inc., in collaboration with HAA , built the fibers for the Gemini Remote Access to CFHT ESPaDOnS Spectrograph (GRACES) instrument. As a result of its involvement in the project, the company gained know-how related to the design and development of optical fiber cables. The company is now developing similar cables for the National Astronomical Observatory of Japan and for postsecondary institutions in the US, including Yale University and MIT .
Dynamic Structures Ltd. ( DSL ), Port Coquitlam, BCDSL designed and built enclosures for a number of observatories around the world, including Gemini and CFHT . The company was subsequently able to translate its expertise and experience in building observatory enclosures to become a world leader in the design and manufacturing of amusement park rides. DSL was selected in 2015by the federal government to design, build and deliver the precision-steel enclosure for TMT and will receive an investment of up to $70million for that purpose (Empire, 2015, n.d. ; ISEDC , 2015).
Institut national d'optique ( INO ), Ville de Québec, QCAs a result of its collaborations with HAA , INO has:
  • helped develop, build and test the optics for GPI (Université de Montréal, 2015).
  • built the calibration unit for Raven Multi-Object Adaptive Optics ( MOAO ).
  • been awarded four contracts for engineering and design services for various NFIRAOS subsystems, including the off axis paraboloid, beamsplitter, instrument selection mirror, and turbulence generator ( PWGSC , 2014a, 2014b, 2015a, 2015b).

According to an INO representative, HAA offered a one year internship to one of INO 's scientists, which allowed them to gain experience in designing adaptive optics and in testing instruments operating at low temperatures.

Projected economic impacts of Canada's participation in TMT and SKA

In 2014, NRC commissioned the Doyletech Corporation to conduct a study to assess the commercial value and potential commercial applications of astronomy-related technologies under development by HAA for the TMT and SKA projects. The study found that technologies being developed for TMT and SKA have the potential to be applied to a number of fields. For example:

  • AO technology for TMT may be used to improve products and services in the areas of sonar and radar, biomedical imaging ( e.g. , MRI and microscopy), ophthalmology and smart-phone cameras (Doyletech, 2014, p. 5). As of 2014, the market for AO technology was approximately US$325billion, increasing to US$470billion by 2019. TMT technologies adapted to various markets could cause firms in these domains to double their sales from approximately US$4.2billion in 2014to US$8.8billion by 2019(Doyletech, 2014, p. 1).
  • Technologies currently being developed for the SKA , such as amplifiers, digital signal processing, PAF s, and dish antennas, can be utilized in the telecommunications, radar, biomedical imaging and aerospace sectors (Doyletech, 2014, p. 6). As of 2014, SKA technologies are applicable to a US$100billion market, increasing to US$137.6billion in 2019(Doyletech, 2014, p. 1). These technologies could enable user companies to increase their sales from approximately US$9.3billion in 2014to US$14.9billion by 2019(Doyletech, 2014, p p. 1–2).

4.1.7Support for universities

Key Finding 20: HAA contributed to the training of students and post-doctoral researchers in Canada.

HAA contributed to the development of an average of 22students (Students and Co-op) and eight research associates annually through temporary employment at NRC . Internal key informants and data indicated that this is an important part of HAA 's talent management strategy, as some of HAA 's current employees were initially hired as students/research associates.

HAA contributes to the training of students in the following manner:

  • Participation of research associates and students in general program activities, including the development of instrumentation: For example, the ALMA Band 3receiver project involved the participation of four co-op students, two graduate students and one post-doctoral researcher. Also, through HAA 's partnerships with universities there are opportunities for students to be involved in other instrumentation projects.
  • HAA staff working as adjunct professors: While data for the entire evaluation period was incomplete, an internal report shows that from January to March 2016, 16 HAA staff had positions as adjunct faculty at three Canadian universities and 11 HAA staff were supervising 18students from four Canadian universities.
  • Providing students with access to national and international telescopes: Data was unavailable to provide a complete picture as to the extent of student use of Canada's telescopes, however, the survey of telescope users showed that during the evaluation period at least nine graduate/PhD students and five post-doctoral researchers applied and received time on one or more telescopes supported by HAA .
  • Providing access to archival data from the CADC : HAA staff encourage student use of archival data and periodically visit Canadian university astronomy departments to make presentations to faculty and students regarding the CADC . Interviews with two Canadian astronomers indicated that CADC archival data was being used for the training of students. One interviewee reported that archival data is being used in the classroom for training students and considered that the access to telescopes and to telescope data is a critical component to recruiting young astronomers to study in Canada.

The evaluation team examined a sample of 15students hired by HAA in order to understand the career path of some students. It was found that some former students have received opportunities as post-doctoral researchers at other institutions and others are professors at universities. Similarly, several past Research Associates are currently employed by HAA , whereas others are professors at Canadian and international universities. Recent former students of HAA have also sought careers outside of astronomy and are positioned in fields such as engineering, finance, programming and software development.

HAA staff expressed concern with how changes to NRC 's 2011Visiting Worker Policy may have created barriers to hiring graduate students. More specifically, provisions in the policy in the areas of IP , publishing, insurance, security and stipends have been viewed by HAA management as limiting their ability to engage with universities for the hiring of students. Further, HAA staff noted that the NRC transformation affected their ability to collaborate with universities as part of instrumentation projects and their ability to hire students.

4.1.8Canada's position in the field of astronomy

Key Finding 21: In certain areas of astronomical research, Canada plays an important role or is considered a world leader. Further, HAA 's activities in astronomical research and, in particular, observatory instrumentation design and development are considered critical to supporting Canada's overall performance in astronomy.

Canada's position in the field of astronomy

The evaluation could not conclusively determine whether there has been a change in Canada's position in the field of astronomy since 2011(the period of the evaluation). However, findings from the survey and key informant interviews do provide some insights on Canada's role, performance and expertise in the field of astronomy over the last five years.

Nearly half of survey respondents (40percent) indicated that Canada's position in the field of astronomy has stayed the same and a quarter indicated that Canada's position has improved.

Key informants generally described Canada as one of the world leaders in the field of astronomy. International expert interviewees noted that Canada "punches above its weight" and "does better than it should" in astronomy given the size of its population and the funding provided for astronomical research. Key informants indicated that Canada has particular expertise or is a leader in areas such as:

  • extrasolar planets (exoplanets)
  • pulsars
  • stellar evolution
  • large astronomical survey research
  • cosmology (including galaxy formation and evolution)
  • astroseismology
  • cosmic magnetism
HAA 's role in supporting Canada's position in astronomy

"Overall, NRC Herzberg's publications stand out at the world level in terms of scientific impact, particularly for its core publications published in the subfield of Astronomy and Astrophysics, which are much more cited on average than similar publications at the world level."

Key informant interviews and results from the survey of telescope users indicate that HAA plays an important role in supporting Canada's performance in astronomy.

Nearly all survey respondents (82percent) "strongly agree" or "agree" that HAA has been an important factor in supporting Canada's performance in astronomy over the last five years. Similarly, key informants note that the Portfolio's role in observatory governance, research and instrumentation projects is a critical factor that supports Canada's performance and leadership position in the field of astronomy.

Some key informants observed that HAA 's General Manager ( GM ) has played a key leadership role by actively participating in observatory governance and representing Canada at various observatory boards and committees, and has been effective in communicating Canadian interests. A few Canadian astronomers and international experts considered the GM 's involvement in the governance and planning for TMT and SKA as one of Canada's main contributions to these projects.

4.2Efficiency and economy

To assess the efficiency and economy of HAA , the evaluation examined stakeholder satisfaction with the application process for time allocation at Canada's national and international observatories; HAA 's delivery of instrumentation projects, observatory initiatives and programs to reduce costs; as well as key risks and factors affecting HAA 's ability to operate efficiently.

4.2.1Efficiency of the time allocation process for observatories

Key Finding 22: The large majority of Canadian astronomers are satisfied with the application process to national and international observatories.

Access to DAO and DRAO

Survey respondents who accessed DAO and DRAO expressed a high level of satisfaction with all the aspects of the application process. More specifically, all of the respondents who accessed DAO were satisfied with the overall efficiency of the application process and only one participant was dissatisfied with the application process for DRAO . While no specific improvements were recommended for the DAO , some survey respondents suggested that HAA provide remote access to the DRAO telescopes. Some respondents expressed concerns about the security screening requirements for foreign nationals which were perceived as a barrier for accessing DRAO telescopes. As noted by HAA management, HAA delivers science-ready data products for the Synthesis Telescope and users usually ask to visit in order to understand the data processing. Additionally, HAA sometimes requires users to come and operate the telescope for safety and staff resources reasons. However, this is not always possible since the NRC is required to follow Treasury Board policies and standards on operational and physical security.

Access to international observatories
Overall efficiency of the application process

Both the survey results and the interviews conducted with Canadian astronomers confirmed that the application process for accessing telescopes is well administered, including the operations of the CanTAC .

Strong levels of satisfaction, with the overall efficiency of the application process, were expressed by survey respondents who applied for time on the CFHT (92percent), Gemini (89percent) and JCMT (79percent) observatories. In the case of Gemini and JCMT , only one survey respondent expressed dissatisfaction with the overall efficiency of the application process.

While the ALMA applicants who participated in the survey expressed a lower level of satisfaction (63percent) when compared with the other international telescopes, it should be noted that only seven percent of these survey respondents expressed a negative view of the application process. Over one quarter (27percent) "neither agree nor disagree" with the statement.

It should also be noted that the application process for ALMA is administered by the Joint ALMA Observatory ( JAO ). Canadian astronomers are, however, involved in the proposal review and time allocation process. Comments from HAA management indicate that the survey results are reasonable given that ALMA is still maturing from an operational perspective and the community is still familiarizing itself with the telescope and the time application/allocation process.

Specific aspects of the application process

As shown in Figure9, a majority of survey respondents have positive perceptions of nearly all aspects of the application process for accessing telescopes.

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (10)

Long description of Figure9

Figure9is a horizontal bar chart which illustrates telescope users' satisfaction with various aspects of the application process to Canada's offshore observatories.

Aspect of application processObservatories
CFHT GeminiJCMT ALMA
The application process for accessing the telescope was merit-based92%89%79%77%
The level of effort and requirements associated with the application process were reasonable100%80%79%60%
Applications to the telescope were processed in a reasonable amount of time96%91%89%63%
I was provided with adequate feedback on the results of my proposal74%78%63%60%
The staff at HAA provided valuable advice that improved the quality of my proposal48%52%37%57%

The survey results also show that satisfaction levels were lowest, for all four international telescopes, in relation to the value of advice provided by HAA staff on proposals. However, most of the respondents who did not agree with this statement answered "Neither agree nor disagree" (8%), "Don't know" (5%) or "Not applicable" (28%).

4.2.2Portfolio indirect costs

Key Finding 23: HAA is operating with a small amount of overhead; the smallest overhead of any NRC Portfolio.

One of the corporate KPI s used to assess HAA 's efficiency is indirect expenditures (overhead) as a proportion of total expenditures. For the period of FY 2012-13to FY 2015-16, HAA 's indirect expenditures accounted for nine percent of total expenditures, which is less than the 12percent target set by HAA Footnote 8. These indirect expenditures do not include the administrative costs associated with services provided by NRC Corporate in support of HAA activities ( e.g. , Human Resources ( HR ), Business Management Support ( BMS ), Administrative Services and Property Management ( ASPM ), and Communications). Adding these expenditures would increase the overhead costs associated with HAA operations. In addition, internal key informants noted that corporate requirements, in particular reporting requirements, decrease the efficiency of HAA .

Key Finding 24: Reporting requirements under the Portfolio/program structure are perceived as "burdensome" and "artificial".

All internal key informants reported that the portfolio/program structure imposed by NRC was artificial and that the formalization of this structure "on paper" through the NRC Program Lifecycle did not result in any real changes in the way HAA actually operates. Reporting requirements associated with the new structure, such as quarterly program reports, was the main issue identified, since these reporting requirements have created overhead which does not add value for HAA .

4.2.3Efficiency in delivering instrumentation projects

Key Finding 25: HAA delivered projects on time, on budget and according to required specifications.

" HAA has been developing instruments for [observatory name] for 20years and is a reliable and good contributor. In the first place, we like to work with them because they're very rigorous. They can deal with deliveries. HAA has excellent engineers. (...) They have expertise and they have one of the top three to five groups in the world for instrument building."

There was a consensus among key informants who collaborated on instrumentation projects with HAA ( i.e. , representatives from observatories, research collaborators and industry representatives) that the Portfolio consistently delivered its projects on time, on budget and according to technical specifications. HAA management and staff were characterized by external collaborators as "team players" who have a strong focus on clients.

The evaluation found that HAA has developed strong project management skills and tools through the delivery of increasingly complex R&D projects that involve large teams of national and international collaborators. HAA also demonstrated its ability to adapt to the various project management requirements of the observatories that contract HAA .

However, when asked about the benefits of the new project management processes and tools implemented by NRC Corporate as part of NRC 's transformation, internal key informants indicated that these tools are not used for the management of projects and felt that they did not add value to HAA 's operations. In fact, they perceived the project management function of the SAP system as a reporting tool that exclusively benefits NRC Corporate. Moreover, internal key informants reported that the requirements imposed by NRC Corporate to enter project data in SAP creates overhead for the Portfolio because an employee has to dedicate a significant amount of his or her time to enter all HAA project information in SAP on a quarterly basis. The Vice-President's Office of Emerging Technologies– National Infrastructure and Future Technologies is aware of these issues and has initiated discussions with HAA management.

4.2.4Key risks and factors influencing the efficient use of resources by HAA

Evidence suggests that the context in which HAA operates has created operational challenges and risks to its ability to achieve its objectives. More specifically, the main factors are: the volatility of the Canadian dollar, challenges associated with common services, and key changes to the general Information Technology ( IT ) environment at the federal government level. This section briefly discusses these factors and their impacts.

Volatility of exchange rates

The decreasing value of the Canadian dollar has had a significant impact on the cost of the G&C portion of HAA 's expenditures. The impact relates primarily to the difference between the rate used to construct HAA 's annual forecast and the rate that is in place at the time an invoice is due. A fluctuation in the exchange rate of, for example, 10percent can mean a change in the cost of over $1million ( CAD ). HAA , with the help of NRC Finance, has implemented controls and mechanisms to plan for and mitigate the impact of these increased costs. For more on this issue, consult Section 2.3.2 of this report.

Challenges associated with common services

As part of the implementation of NRC 's Strategy 2013-2018, many administrative functions ( e.g. , HR , finance, communication, facility management, business management, procurement and IT support) that were previously under the purview of institutes were centralized. An extensive evaluation of NRC Common Services was not conducted as part of this evaluation. Still the evaluation did observe some inefficiencies with some of the services provided to HAA .

Key Finding 26: Operational challenges were encountered in the areas of procurement, hiring and IT support.

Key Finding 27: HAA requires access to specialized IT desktop support for UNIX and Mac. However, NRC Knowledge and Information Technology Services ( KITS ) desktop support is provided almost exclusively for Microsoft Windows.

Internal key informants indicated that the centralization of NRC Corporate services resulted in a reduction in the level and quality of many administrative functions. Key challenges most frequently identified by internal key informants and in HAA planning documents include:

  • Procurement services: Increased demand for procurement services, particularly in the short term due to expanding workloads associated with the TMT project, was identified as a key risk which could limit HAA 's ability to deliver instrumentation projects. This could result in potential project delays and adverse impacts on the Portfolio's reputation and its ability to obtain future instrumentation projects. As a result of the NRC centralization, FTE s supporting the procurement process were reduced from four to two. To meet its procurement needs, HAA contracted a local Public Works and Government Services Canada ( PWGSC ) contracting officer on a part-time basis to facilitate the later stages of the procurement process.
  • Hiring: The average time to complete a staffing action more than doubled between FY 2012-13and FY 2015-16, increasing from 69days to 155days. Internal key informants reported that the length of the staffing process has been a contributing factor to at least one candidate dropping out of the hiring process. However, this increase was not found to be a major impediment to the attraction of highly qualified personnel, likely due to HAA 's reputation. HAA continues to attract world-class researchers in the field of astronomy.
  • IT Desktop support: KITS support is provided almost exclusively for Microsoft Windows users, whereas almost all HAA researchers are UNIX and Mac users. According to HAA researchers, the UNIX system is customary for research in astronomy since software used and developed by and for the astronomy community is written for UNIX machines (Macs natively run a standard variant of UNIX). While KITS has recently created a group to provide desktop support for UNIX and Mac users, internal key informants did not mention having access to the services provided by this group during the evaluation period.

Evidence suggests that if these challenges are not addressed they could impact the Portfolio's ability to achieve its mandate moving forward.

General IT environment– Shared Services Canada ( SSC ) and NRC cyber-intrusion

Key Finding 28: The transfer of HAA 's IT infrastructure to Shared Services Canada has limited the Portfolio's ability to provide data management services to its clients (the astronomy community).

In July 2012, HAA 's IT infrastructure (servers and networking) for providing access to Canadian astronomy data transitioned to a new centralized government department, SSC . The impact of the transfer of CADC resources to SSC has been characterized by HAA as "serious" in terms of their ability to plan, implement and procure new equipment and network capacity. Issues with the service levels provided by SSC are not unique to HAA , but are consistent with the Office of the Auditor General report published in the Fall of 2015which details how SSC has had difficulty maintaining IT service expectations with its clients across the federal government ( OAG , 2015). The NRC cyber intrusion in 2014further challenged the delivery of these services (since SSC and K IT S efforts were focused on issues related to the cyber intrusion).

The evaluation found two key issues that were faced by HAA , during the period covered by the evaluation, with regard to providing access to telescope data:

  • Need for IT Equipment Upgrades: According to internal documentation, the collaboration process with SSC to procure IT equipment upgrades posed challenges as communications were "sporadic or non-existent." The need to replace outdated servers and storage systems equipment placed NRC at risk of non-performance against contracted service obligations and affected the overall quality of service provided to its clients. Rather than continue to pursue SSC to provide IT infrastructure upgrades to support the CADC , HAA addressed this issue through the establishment of a development contract with Compute Canada ( CC ). This project is currently underway.
  • Need for High Speed Networking to Provide Access to Users: The previous 2011evaluation concluded that " CADC 's services are not optimized due to lack of access to required high-speed network that restricted the service levels that the CADC could provide its clients"Footnote 9. As of July 2016, the network has not yet been upgraded from one gigabit per second to ten gigabits per second in order to match the speeds available to researchers using other networks. Currently, there is no agreement between KITS , SSC , and HAA on how to procure a solution that satisfies all parties. According to internal key informants, the need for a network improvement is an ongoing critical issue for HAA . Internal data reveals that limited network capabilities at the CADC have negatively impacted HAA 's ability to provide access to telescope data. The demand for access to CADC 's data often resulted in over-saturated networks, causing network failures and interruptions of the services to users and disrupting services of other NRC enterprise users. As an interface with the world astronomy community, inconsistency in the CADC 's service levels presents a potential risk to the reputation of NRC .

Additionally, the cyber-intrusion and its corollary, the creation of a new IT infrastructure (referred to as the "green zone"), was found to add a layer of complexity to the functionality of the NRC IT environment. In fact, HAA staff felt that having to navigate between the IT infrastructures with two laptops has made simple tasks, such as time entry and service requests to NRC Common Services, burdensome. Additionally, according to HAA management, the provision of Green laptops at DRAO was done without any prior consultation and "forced the observatory to spend many hours to design and test new radio frequency interference ( RFI ) containment boxes for those machines". According to this manager, the cost of the RFI boxes far exceeds the cost of the laptops which, moreover, are not useful for HAA research activities.

When describing the current IT environment at NRC an HAA researcher mentioned that "it is hard to believe that such a highly regarded organization was so inefficient in terms of providing basic IT support". According to the key informant, this situation could create a major barrier to HAA 's ability to retain world-class researchers.

Recommendation 5: HAA , in collaboration with KITS , should continue to undertake efforts to find a solution related to the IT equipment upgrades and high speed networking capacity issues at the CADC .

4.2.5Efficient use of funding by administrators of international telescopes

Key Finding 29: Observatory administrators are implementing initiatives and programs to reduce costs and improve operational efficiency.

Observatory annual reports provide a number of examples of initiatives undertaken by observatories to reduce costs and improve overall efficiency. Examples of these initiatives include:

  • Improved energy efficiency and reduced energy costs: In 2013, a photovoltaic system (for solar power) was installed on the roof of the CFHT headquarters building in Waimea to offset approximately 40% of energy use. Further, conventional fluorescent lamps were replaced with LED lamps throughout the building ( CFHT , 2013, p. 20).
  • Gemini Transition Program: Following the withdrawal of the UK and a reduction of approximately 20% ($7.4million) in its operating budget, Gemini launched a Transition Program in 2013to introduce cost savings activities while sustaining operations. According to Gemini's 2015Annual Report, the observatory has achieved spending reductions of $6.5million from its 2012budget. These savings are the result of three activities: reduction in staff (from 212 FTE s in 2011to 178 FTE s in 2015); reduction in non-labour expenses (such as supplies, travel and computers); and implementation of projects that support operations with a smaller staff or achieve additional yearly savings (including reducing data archive costs, increasing visitor observing, and energy savings) (Gemini Observatory, 2015, pp. 18–21).
4.2.6 HAA 's ability to influence decision-making processes in the governance of Canada's offshore telescopes

Key Finding 30: HAA staff participate in various observatory boards and committees, and are consequently directly involved in the decision-making processes of international telescopes supported by Canada.

As mentioned in Section 3.1.1, HAA has been a member of numerous boards and committees across the five observatories reviewed by this evaluation. HAA was a member of the Board and the Finance/Budget Committee for all five observatories. In fact, key informants noted that HAA 's GM and an HAA Director were Chairs of the Finance/Budget Committees at Gemini and CFHT . These committees are responsible for approving and reviewing observatory budgets. Further, HAA 's GM was Chair of the SKAO Board and is currently the Vice-Chair of the Thirty Meter Telescope International Observatory (TIO). HAA is also a member of the Audit Committees at the CFHT , TIO and SKAO and, as such, is involved in examining financial statements and reviewing external audit reports. As a result, HAA is directly involved in the decision-making processes regarding observatory operations, planning and finances. Finally, key informants pointed to HAA representatives on the ALMA Board, noting that:

  • the HAA representative on the ALMA Board is currently servicing as Vice-Chair (an important role for setting and controlling the agenda and workings of the Board).
  • an HAA representative is the Chair of the ALMA Budget Committee.

5.0Conclusions

Overall, the evaluation found that HAA is aligned with the LRP2010 which ensures that the Portfolio's activities and investment priorities meet the needs of the Canadian astronomy community. Additionally, HAA is perceived as playing a critical role in representing the interests of the Canadian astronomy community internationally through its involvement in existing and future observatories. HAA 's facilities at DAO and DRAO are unique in Canada, providing core capabilities and expertise in observatory instrumentation and infrastructure development.

HAA has performed well and has made significant impacts by supporting Canada's offshore observatories as well as other telescopes around the world. This support has been mainly through HAA 's involvement in the governance of international observatories, the provision of financial support and the development of world-class instruments and technologies. HAA has collaborated with industry on various instrumentation projects, resulting in positive impacts for partnering companies. While HAA has been effective at meeting its objectives during the evaluation period, evidence suggests that the Portfolio has encountered operational challenges which may impact HAA 's ability to fulfill its mandate.

The areas of greatest concern are related to IT . In particular, limited network capabilities at the CADC have impacted HAA 's ability to provide access to telescope data. Further, as the observational data collected by observatories becomes increasingly complex, particularly with next-generation telescopes, the need for HPC and improved data management capabilities was identified as a critical issue that must be addressed if the CADC is to remain a key player in astronomical data archiving and processing. The evaluation noted that since the transfer of HAA 's IT infrastructure to SSC , HAA has experienced challenges in its ability to plan, implement and acquire new equipment and networking capacity. Additional challenges included procurement delays, lengthening timelines for staffing actions, and limitations in IT desktop support.

Through its ongoing involvement in next generation observatories, including SKA and TMT , HAA appears to be well positioned to continue to achieve impacts in the future.

6.0Management response

RecommendationResponse and Planned Action(s)Proposed Person(s) ResponsibleTimelinesMeasure(s) of Achievement
The Vice-President's Office ( VPO ) of Emerging Technologies– National Infrastructure and Future Technologies, in consultation with HAA , should consider establishing a formal consultation process to allow the Canadian astronomy community to provide strategic advice on the Portfolio's scientific activities and priorities.Accepted. NRC will investigate appropriate mechanisms for obtaining advice on a regular basis from the astronomy community and will develop a proposed process early in FY 2017-18.VP
GM
May 2017A consultation mechanism allowing the community to provide feedback on HAA strategic directions is developed and implemented.
HAA should continue to work with partners to expand outreach and awareness activities.Accepted. HAA will continue to work in concert with partners to expand outreach activities. A new model to optimise use of the Centre of the Universe will be explored.GM June 2017A plan for local outreach to the community is put in place.
HAA , in collaboration with other NRC Portfolios and NRC Business Management Support ( BMS ), should explore opportunities to leverage competencies and astronomy-related technologies.Accepted. HAA with support from the VPO of Emerging Technologies– National Infrastructure and Future Technologies and BMS will engage into discussions with the broader NRC community and ensure that competencies and intellectual property held at HAA are known and shared.GM October 2017Discussions have taken place and competencies and IP originating at HAA are introduced to other NRC Portfolios.
HAA should identify the reasons why Canadian PI s are not allocated observing time commensurate with Canada's financial support to ALMA operations and take the appropriate measures identified, as needed.Accepted. HAA will work with the Canadian astronomy community and with the National Science Foundation to understand why the allocation of ALMA time to Canadian astronomers within the allocation of ALMA time to the North American Consortium has not been proportional to Canada's share of the North American Contributions and will attempt work to remove any impediments to Canadian access.GM September 2017(for Cycle 5 ALMA proposals)Impediments to Canadian access are identified, and an action plan to overcome these is developed and implemented, as needed.
HAA , in collaboration with KITS , should continue to undertake efforts to find a solution related to the IT equipment upgrades and high speed networking capacity issues at the CADC .Accepted. NRC will continue to work to find a solution to ensure that CADC services to clients are not impacted.GM
CIO
September 2017Discussions with KITS and SSC have taken place, and solutions with appropriate timelines are identified.

AppendixA– HAA Logic model

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (11)

Long description of the HAA logic model follows

The diagram shown in this AppendixA is a logic model for HAA . The logic model shows the progression from portfolio inputs, through activities to outputs and finally to three progressive levels of expected outcomes (immediate, intermediate, and ultimate).

The Logic Model is divided into 6progressive levels which are listed below. Each level feeds into the following level.

  1. Level 1: Inputs
  2. Level 2: Activities
  3. Level 3: Outputs
  4. Level 4: Immediate outcomes
  5. Level 5: Intermediate outcomes
  6. Level 6: Ultimate outcomes

Level 1: Inputs

Inputs are demonstrated through the following indicators:

  1. 1.1 NRC Financial Resources (A-base, transfer payments, other revenues)
  2. 1.2 NRC Herzberg employees
  3. 1.3 NRC Herzberg Large Scale Research infrastructure and equipment
  4. 1.4 NRC Corporate services and SSC support and resources
  5. 1.5Students, Post-docs, visiting workers, volunteer workers
  6. 1.6Intellectual property

Level 2: Activities:

The second level of the diagram is a list of activities, including:

  1. 2.1Maintain scientific, engineering and technical capabilities to participate in the operations and the development of observatories to serve Canadian astronomers
  2. 2.2Canadian Astronomy Data Centre
  3. 2.3Assist with observational and post-observational data analysis
  4. 2.4Provide Canadian astronomers, including students and post-doc researchers, with access to national and international observatories
    1. 2.4.1Administer the time allocation process for Canadian researchers at national and international observatories
    2. 2.4.2Operate and maintain Canadian observatories
  5. 2.5Development of leading-edge instruments through R&D projects with the scientific support of HAA astronomers
  6. 2.6Know-how and technology transfer to Canadian industry
  7. 2.7Administration and governance of current and future international observatory agreements
    1. 2.7.1Administer Canada's share of the four telescopes
    2. 2.7.2Participate in telescope governance and scientific meetings
    3. 2.7.3Provide leadership in developing new opportunities in next generation observatories
    4. 2.7.4Consultation with the Canadian astronomy community
  8. 2.8Public outreach

Level 3: Outputs

The third level of the diagram is a list of outputs:

  1. 3.1Science data products
  2. 3.2Assistance provided to Canadian astronomers
  3. 3.3Access to national and international observatories (Canadian time is allocated on merit in a timely manner)
  4. 3.4Access to laboratories, facilities and instruments
  5. 3.5High quality data are produced for astronomers, students, and post-doc researchers
  6. 3.6Novel instruments and technologies for telescopes
  7. 3.7Benefits to Canadian industry
    1. 3.7.1Canadian industry has increased opportunities to participate in advanced scientific projects and benefit from contracts
    2. 3.7.2Canadian industry has new or improved technological capabilities
  8. 3.8Canada's interests in international agreements are reflected in the board decisions and investments are managed in a diligent manner
  9. 3.9Canada's participation in next generation observatories is secured
  10. 3.10Increased public awareness and understanding of science and astronomy

Level 4: Immediate outcomes

The fourth level of the diagram describes immediate outcomes which are intended to take place between 1to 5years.

Immediate outcomes include:

  1. 4.1Students and post-doctoral researchers are trained on world-class facilities and technologies
  2. 4.2Publication of high impact science by Canadian astronomers
  3. 4.3Increased competitiveness of Canadian industry through participation in telescope contracts
  4. 4.4Increased international research collaborations for Canadian astronomers

Level 5: Intermediate outcomes

The fifth level of the diagram describes intermediate outcomes which are intended to take place between 5to 10years.

Intermediate outcomes include:

  1. 5.1Retention of Highly Qualified Personnel ( HQP ) in Canada
  2. 5.2Canada plays a prominent role in international scientific endeavor
  3. 5.3Growth of the Canadian industry resulting from sales and contracts and technology spillover

Level 6: Ultimate outcomes

The sixth and final level describes ultimate outcomes which are intended to take place between 10to 20years.

Ultimate outcomes include:

  1. 6.1Important knowledge is acquired about the universe
  2. 6.2Canada's position is enhanced among the world's leaders in astronomy

AppendixB– Methodology

This section presents a detailed description of the evaluation methodology, including the evaluation design and approach, the challenges and limitations encountered during the evaluation, and the data collection methods.

Evaluation Design

The evaluation was carried out in accordance with the NRC 's approved evaluation plan and TBS policies. The last evaluation of NRC - HAA took place in 2011. The timing of the evaluation was also determined by the legal requirements of the Financial Administration Act which indicate that the evaluation of HAA 's contribution to international observatories needed to be completed no later than November 2016.

The questions addressed by the evaluation are presented in Table7. The evaluation questions were developed following a review of key program documents and in consultation with HAA management and the VP , Emerging Technologies. It should be noted that a complete and in-depth assessment of the performance of NRC 's internal services was not within the scope of the evaluation.

Table7: Evaluation Issues and Questions
Evaluation IssuesEvaluation Questions
Relevance– Continued need for program
  1. To what extent is the HAA Portfolio aligned with the current needs of Canadian astronomy researchers?
  2. To what extent is the current delivery model for the administration and operations of Canada's telescopes appropriate to meet the needs of the Canadian scientific astronomy community?
    • Are there alternative delivery models that could better serve the needs of the Canadian scientific astronomy community?
    • What are the conditions required for such an alternative model to meet Canada's legal, regulatory and operational requirements?
Relevance– Alignment with government and NRC priorities
  1. To what extent is the HAA Portfolio aligned with federal priorities and NRC strategic objectives?
    • What have been the impacts of NRC 's transition to a Research & Technology Organization ( RTO ) model on HAA 's activities and strategic priorities?
Relevance– Alignment with federal roles and responsibilities
  1. To what extent are HAA 's activities and strategic outcomes consistent with federal roles and responsibilities?
Performance– Effectiveness
  1. To what extent has HAA been successful at providing Canadian astronomers with access to:
    • national and cutting-edge international telescopes; and
    • science data products ( i.e. , CADC )?
  1. What have been the scientific impacts stemming from the national and international telescopes operated and administered by HAA ?
  1. What have been HAA 's contributions to the development of novel instruments and technologies in support of Canada's current and future telescopes?
    • What have been the industrial impacts of the novel instruments and technologies developed by HAA ?
  1. To what extent and in what ways has HAA supported Canadian universities with regards to the training of students and post-doctoral researchers?
  1. To what extent has HAA contributed to strengthening Canada's position among the world leader in astronomy?
Performance– Efficiency and Economy
  1. To what extent is HAA administered in a cost-effective and efficient manner?
    • What have been the factors that facilitated or hindered HAA 's ability to manage its resources in a cost-effective and efficient manner?

The evaluation of HAA was conducted by an independent evaluation team from the NRC Office of Audit and Evaluation. The work of the evaluation team was supported by an Evaluation Advisory Committee ( EAC ) to provide advice and input on key deliverables.

In order to maximize the possibility of generating useful, valid and relevant evaluation findings, the evaluation used a mixed methods approach, allowing for triangulation ( i.e. , convergence of results across different lines of evidence) and complementarity ( i.e. , using complementary research methods to provide different perspectives when examining various facets of complex issues). The methods employed in the study as well as challenges and limitations are described below.

Methods

This evaluation employed both qualitative and quantitative research methods, including:

  • Literature and document review
  • Analysis of administrative and performance data
  • Survey of Canadian telescope users
  • Key informant interviews
  • Case studies

Literature and Document Review

Internal and external documentation was reviewed, synthesized and integrated into the evaluation to provide contextual and historical information, and to complement other lines of evidence in assessing relevance and performance. Internal documents included HAA strategic planning reports, business plans and implementation plans, as well as NRC Departmental Performance Reports ( DPR ) and Reports on Plans and Priorities ( RPP ). External documents included CASCA publications, such as the LRP2010 and LRP2010 Mid-Term Review, as well as publications and reports prepared by Canada's offshore telescopes.

Analysis of Financial, Administrative and Performance Data

Financial, administrative and performance data were reviewed to provide information on program outputs and impacts, and to contribute to the analysis of resource utilization. Administrative and performance data were provided by HAA staff while financial data was provided by NRC Finance Branch.

Data consulted included subscription and publication rates for Canada's offshore observatories, Portfolio expenditures and revenues, as well as HR statistics.

Survey of Canadian Telescope Users

A survey of users of Canada's national and international observatories was conducted. The survey sample included Canadian astronomers who applied for time on the telescopes between 2011and 2015. More specifically, the survey targeted Canadian astronomers with a faculty position in a Canadian postsecondary institution, students at a Canadian postsecondary institution, and post-doctoral researchers. The survey population did not include HAA researchers.

The survey assessed users' experience in terms of access and collected information about Canada's performance in astronomy since 2011. More specifically, respondents were asked a series of questions to assess the time allocation process administered by the observatories, to describe the scientific impacts that resulted from their access to the telescopes, and to identify Canada's contributions to the field of astronomy over the last five years.

The web survey was offered in both French and English and was administered by NRC 's Communications and Corporate Relations Branch ( CCRB ) using web-based software (FluidSurveys). The survey was available for three weeks. An initial email invitation and two reminder emails were sent by the HAA General Manager during the survey period.

Overall, invitations were sent to a total of 306individuals, of which 11did not identify themselves as direct users of national and international ground-based telescopes and were therefore not included in the survey sample. As a result, the final sample was 295. A total of 95respondents completed the survey, for a response rate of 32percent.

A survey report analyzing the results was prepared. Where possible, the results of the survey were compared to those obtained in the survey conducted as part of the previous evaluation in 2011.

It should be noted that results from the survey indicate that one of the questions was not formulated clearly. When asked about whether HAA staff provided them with valuable advice when preparing their proposals for observing time, most respondents who did not agree with this statement answered the following three options: "Neither agree nor disagree", "Don't know", or "Not applicable". The survey question should have asked about the support provided not only by HAA but by the observatories as well. In this regard, it was brought to the attention of the evaluation team during the analysis of survey results that the advice is usually provided directly by the observatories or by external committees. In the case of CFHT , for example, HAA provides support to the observatories to employ Canadian Resident Astronomers with the understanding that such staff provide an informal point of contact for Canadian applicants and users of the CFHT . For Gemini, the feedback is provided by the National Gemini Office that is administered by HAA employees, as per the Gemini partnership agreement.

Key Informant Interviews

Internal and external stakeholders were consulted to collect qualitative evidence with regards to the relevance and performance of NRC - HAA . The information gathered through the qualitative, semi-structured interview process was based on personal experience, opinions and expert knowledge. This information was important for contextualizing performance data and other statistics.

Interviews were conducted either in-person or by telephone. Each interview lasted approximately 30to 90minutes and was conducted using an interview guide. The interview guides were adjusted to each key informant group, ensuring that questions were relevant to each group. All key informants received a copy of the interview guide in advance of the interview.

Overall, a total of 31interviews were conducted with NRC and NRC - HAA staff, external stakeholders, representatives from Canada's offshore observatories, and international experts. Interviewees were selected in consultation with HAA management and the EAC . To encourage participation, the HAA General Manager sent an email to potential interviewees inviting informing them of the evaluation and inviting them to participate. The evaluation team followed up with key informants to confirm their participation and to schedule an interview.

A summary of the key informant groups consulted as part of this evaluation is presented in Table8.

Table8: Summary of key informant interviews
Interviewee categoryn
NRC - HAA management and staff13
External members of the astronomy community6
Other federal government departments and agencies2
Telescope directors4
Industry partners3
International astronomy experts3
Total31

Case Studies

A series of four case studies were conducted as part of the evaluation to illustrate the types of projects and activities undertaken by HAA in support of international observatories, as well as to gain a better understanding of the Portfolio's role in astronomy research and instrumentation development and the impacts resulting from HAA projects, including identifying factors and conditions that facilitate or impede the achievement of impacts. The case studies involved a review of project-level documents and interviews with project proponents, including HAA staff, representatives from international observatories, and representatives from postsecondary institutions and industry involved in the project.

A brief report summarizing the findings was prepared for each case study project.

The four case studies included in the evaluation, and the number of key informants consulted, is presented in Table9.

Table9: Summary of case study interviews
Case studyn
N ALMA Band 3Receiver5
Gemini Planet Imager ( GPI )4
Pan-Andromeda Archeological Survey ( PAndAS )4
PWIDAR Correlator for the EVLA 5
Total18

Challenges and limitations

The evaluation encountered certain limitations and challenges that must be taken into consideration when interpreting the findings. These challenges and limitations are common in most evaluations conducted within the federal government. In order to alleviate any adverse impacts on the evaluation findings, various mitigation strategies were identified. Table10 presents the limitations and challenges, their potential impact on the project, as well as the mitigation strategies used to minimize their effects.

Limitations and challenges

The evaluation was not able to provide an assessment of the level of satisfaction of CADC users.

The evaluation identified some limitations to the reliability of NRC Corporate KPI s ( i.e. , indirect costs as proportion of total expenditures).

Impact on evaluation project/findings

Due to limited resources, the evaluation did not survey or interview CADC users to quantify and qualify levels of satisfaction with services.

Data on the Portfolio's indirect expenditures do not include administrative costs associated with services provided by NRC Corporate ( e.g. , HR , BMS , ASPM , and Communication).

Mitigation strategies

The evaluation relied on data provided by the Portfolio as well as perspectives of internal and external key informants, including observatory partners and Canadian astronomers, who use CADC services.

The reliability issues associated with the NRC corporate KPI s were documented to ensure that the evaluation findings are interpreted with caution.

AppendixC: Overview of international telescopes supported by Canada

Table11: Overview of Canada's Offshore Telescopes
Telescope:CFHT JCMT GeminiALMA TMT
Location:Summit of Maunakea, HawaiiSummit of Maunakea, Hawaii(North) summit of Maunakea, Hawaii
(South) Cerro Pachón, Chile
Chajnantor Plateau, Atacama Desert, ChileSummit of Maunakea, Hawaii (Planned location; construction not completed)
Type / Specialization:Optical-InfraRed (3.6-meter)Sub-millimeter (15meter)Optical-InfraRed (twin 8-meter)Millimetre & Sub-millimetreOptical ((near-ultraviolet to mid-infrared (30-meter)
Start of Funding Agreement:19741989199320082015
Date Telescope became operational197919872000/ 2001 (North/ South)2013Not yet constructed
Canada's share42.5%0% (Formerly was 25%)19.88%Footnote 102.8% (7.25% of North American share 37.5%)TBD : Estimated 15%
PartnersFrance (42.5%) and the State of Hawaii (15%)No longer applicableUnited States: 69.84%
Canada: 19.88%
Brazil: 6.96%
Argentina: 3.32%
Europe (37.5%)
North America (37.5%)
East Asia (25%)
In cooperation with the Republic of Chile
United States (25%)
Japan (20%)
China (10%)
India (10%)
University of Hawaii (7.5%)
15% is currently unsubscribed and is available for additional partners
Executing AgencyCFHT Corporation (a not-for-profit corporation)Currently the East Asian Observatory ( EAO ).
Formerly operated by United Kingdom's Science and Technology Facilities Council.
The National Science Foundation ( NSF ) is the executive agency. However, the Association of Universities for Research in Astronomy ( AURA ) manages the organization as authorized through an agreement with the NSF .The Joint ALMA Observatory ( JAO ) provides the unified leadership and management of the construction, commissioning and operation of ALMA .TMT International Observatory
Duration of agreementThe current agreement began February 1, 2015and continues until January 2019. Although there is no fixed termination date; dissolution is provided for in by-laws.Canada withdrew effective September 2014.Current agreement began January 1, 2016and lasts until December 31, 2021.
The prior agreement was scheduled to end December 31, 2015.
The ALMA Trilateral Agreement signed between the NSF , the National Institutes of Natural Sciences of Japan ( NINS ) and the European Southern Observatory ( ESO ) expires no sooner than December 31, 2021.
NRC and NSF have entered into a Contributions Agreement ( CA ) reflecting Canada's commitments under the NRC - NSF MOU signed in 2003. The current term of the CA extends to December 312020, with NRC notice of intent to extend or terminate the CA provided to the NSF by December 31, 2019.
TBD
Annual operating budgetAs of 2014, CFHT 's total budget is approximately $7.2million and includes the following:
  • Staffing: $4.9million
  • Base facilities and operations: $620,485
  • Observatory facilities and operations: $618,684
  • General administrative expenses: $437,771
  • Transfer to reserve: $390,000
  • Science: $81,215
  • Instrumentation: $79,634
  • Outreach: $55,516
Not AvailableAs of 2015, Gemini's total budget is approximately $33.5million and includes the following:
  • Operations and maintenance: $31.3million
  • Instrument Development Fund: $2.2million
  • Facilities Development Fund: $46,564
(All figures in USD )
  • 2015total: $69.2million
  • Chilean operations (Joint ALMA Observatory):
    $39.3million
  • For the above two totals, the North American share is 37.5%
  • Offsite expenses (North America): $10.9million
Not Applicable
Total employees~ 44Not Available~ 165Not AvailableNot Applicable

Note: SKA is not included in the table as the project is still in its early stages and not enough information is available.

AppendixD: Evaluation framework

Evaluation questionDocument reviewData reviewKey informant interviewsCase studiesSurvey of telescope users

1. To what extent is the HAA Portfolio aligned with the current needs of Canadian astronomy researchers?

2. To what extent is the current delivery model for the administration and operation of Canada's telescopes appropriate to meet the needs of the Canadian scientific astronomy community?

2.1Are there alternative delivery models that could better serve the needs of the Canadian scientific astronomy community?

2.2What are the conditions required for such an alternative model to meet Canada's legal, regulatory and operational requirements?

NoNoNo

3. To what extent is the HAA Portfolio aligned with federal priorities and NRC strategic objectives?

3.1What have been the impacts of NRC 's transition to a Research & Technology Organization ( RTO ) model on HAA 's activities and strategic priorities?

4. To what extent are HAA 's activities and strategic outcomes consistent with federal roles and responsibilities?

NoNoNo

5. To what extent has HAA been successful at providing Canadian astronomers with access to national and international telescopes and science data products ( i.e. , CADC )?

NoNo

6. What have been the scientific impacts stemming from the national and international telescopes operated and administered by HAA ?

No

7. What have been HAA 's contributions to the development of novel instruments and technologies in support of Canada's current and future telescopes?

7.1What have been the scientific and industry impacts of the novel instruments and technologies developed by HAA ?

8. To what extent and in what ways has HAA supported Canadian universities with regards to the training of students and post-doctoral researchers?

No

9. To what extent has HAA contributed to strengthening Canada's position among the world leaders in astronomy?

No

10. To what extent is HAA administered in a cost-effective and efficient manner?

10.1What have been the factors that facilitated or hindered HAA 's ability to manage its resources in a cost-effective and efficient manner?

AppendixE: Bibliography

Footnotes

Footnote 1

The international agreement to operate the Gemini observatory requires each partner country to establish a national office to participate in the observatory's operations. In Canada, the National Gemini Office ( NGO ) provides support to the Canadian astronomy community regarding the observatory's proposal process, capabilities and instrumentation.

Return to footnote 1 referrer

Footnote 2

To meet its obligations each FY , NRC usually exchanges funding either through internal adjustments or through supplementary estimates. For example, NRC 's Departmental Performance Report ( DPR ) for 2014-15indicates that $2.2million in additional funding was provided through the supplementary estimates process and an internal transfer from the NRC Transfer Payment Program. Specifically, $1.8million in the Portfolio's operating funds was converted into contribution funds and approximately $400,000was transferred from NRC - IRAP .

Return to footnote 2 referrer

Footnote 3

After the NRC Transformation, the title of Portfolio heads was changed from Director General ( DG ) to General Manager ( GM ).

Return to footnote 3 referrer

Footnote 4

After approximately one year, the proprietary access period will end and the "telescope data product" will become available for use by other astronomers. Making archival data available increases the value from past observations allowing them to be reused, potentially for a completely different purpose.

Return to footnote 4 referrer

Footnote 5

The Centre of the Universe was the public interpretive centre for the DAO that was regularly open to the public between May and September.

Return to footnote 5 referrer

Footnote 6

Additional information regarding the status of the TMT project is available on the TMT Organization website

Return to footnote 6 referrer

Footnote 7

To assess an observatory's impact, the analysis relies on the median impact per paper rather than the average impact per paper, as the latter can be skewed by a small number of very high impact papers. It should be noted that data on impacts are not normalized.

Return to footnote 7 referrer

Footnote 8

It should be noted that a review of NRC Corporate KPI data revealed that HAA has an average staff utilization rate of 96percent for the period 2012-13to 2015-16, which is much higher than other NRC portfolios. While HAA staff utilization rates appear to be unusually high when compared to other NRC portfolios, the evaluation did not conduct a thorough assessment of the reasons for this discrepancy.

Return to footnote 8 referrer

Footnote 9

The 2011evaluation further noted that the research standard of Canadian universities is 10 Gb/s , however, as of July 2013the CANARIE network connecting the Canadian research community is capable of up to 100 Gb/s (CANARIE, 2013).

Return to footnote 9 referrer

Footnote 10

This proportion is Canada's current share as of January 1, 2016. It should be noted that partner shares in Gemini have changed over the evaluation period.

Return to footnote 10 referrer

Evaluation of NRC Herzberg Astronomy and Astrophysics (HAA) Portfolio (2024)

References

Top Articles
Latest Posts
Recommended Articles
Article information

Author: Rubie Ullrich

Last Updated:

Views: 6434

Rating: 4.1 / 5 (52 voted)

Reviews: 83% of readers found this page helpful

Author information

Name: Rubie Ullrich

Birthday: 1998-02-02

Address: 743 Stoltenberg Center, Genovevaville, NJ 59925-3119

Phone: +2202978377583

Job: Administration Engineer

Hobby: Surfing, Sailing, Listening to music, Web surfing, Kitesurfing, Geocaching, Backpacking

Introduction: My name is Rubie Ullrich, I am a enthusiastic, perfect, tender, vivacious, talented, famous, delightful person who loves writing and wants to share my knowledge and understanding with you.