## A Report on the 2012 NSERC Discovery Grants Results for Toronto Math

Fifteen1 faculty members from the Department of Mathematics at the University of Toronto submitted proposals to the 2012 NSERC Discovery Grants competition. Of these, one was a first time applicant (En), two (Ga, Ia) applied after a successful appeal of 2011 results, and one (Cd) was an appellant whose appeal was denied but could reapply because the 2011 award was for zero dollars. The first table below shows the 2012 results (in thousands of dollars per year) with 2010, 2011 award amounts for those researchers. The second table shows similar data for Toronto mathematicians in the 2011 competition, including the amounts for researchers (1d, 2d, 3d, 4d) whose appeals of 2011 results were rejected. The average for 2012 awards to Toronto mathematicians was 153% the average for 2011.

### Average Grant Amount (Toronto Math)

• 2006: \$27k/y • 2007: \$26.3k/y
• 2008: \$26.5k/y • 2009: \$25k/y
• 2010: \$25.3k/y • 2011: \$19.3k/y
• The experience of 2011 appellant Ia is also a bit strange. After a long run of celebrated research funded at the \$40k/y level, this researcher’s funding level was dropped in 2011 to \$15k/y. That outcome was successfully appealed and the 2012 outcome was \$35k/y. • The 2011 appeals by 1d, 2d, 3d, and 4d were all turned down so these researchers are locked in at relatively low funding levels for the next five years. • NSERC deviated from standard policy (a “pilot program”) in their handling of the 2011 appeals of Toronto mathematicians. Toronto’s appeals were evaluated by multiple appeals advisers while appeals from other universities were evaluated by one. There is evidence2 showing that Toronto appeals were denied even when one of the appeals advisers advocated for granting the appeal. ## Consistent Results on Appeals Cases show 2011 was Anomalous Proposals by three Toronto researchers were evaluated in both the 2011 and 2012 competitions. The outcomes for these proposals provide a comparison3 between the accuracy of the merit evaluations by the 2011 and 2012 Evaluation Groups and bin-to-funding assignment by the Executive Committee and NSERC staff. Here is the data, including the percentage adjustment from 2011 to 2012: These three cases provide further evidence, consistent with the message in the public statement signed by over 300 Canadian researchers, that the 2011 evaluations were anomalous. Despite the consensus opinion from the Canadian math/stats community, a public message from a majority of the 2011 Evaluation Group, and advice from top administrators that the 2011 anomalies required an altered appeals process, NSERC chose not to reevaluate the scientific merit of proposals when considering whether to grant or deny an appeal. The grounds for a successful appeal required evidence of administrative errors; evidence of error in merit evaluation was not considered germane. The Math-NSERC Liaison Committee is collecting data from department chairs about the 2012 competition for Section 1508. It might turn out that 2012 will be viewed as more consistent with expectations, a more accurate evaluation compared to 2011. This would be encouraging. However, the unsuccessful 2011 Toronto “pilot program” appellants (researchers 1d, 2d, 3d, 4d) face the next five years with inadequate funding to support their research programs. Footnotes: 1. The names of the faculty members are suppressed. The 2012 applicants will be referenced with codes A, B, C, …, H; 2011 applicants (without successful appeal) will be referenced using 1, 2, …, 7. The appended small case letters indicate whether the researcher had an appeal granted (a), an appeal denied (d), or was a new applicant (n). 2. This evidence is contained in the reports of the appeals advisers provided to the appellants by NSERC and also in documents obtained by some of the appellants through formal requests under the Access to Information Act. 3. It would be useful to know the data (number, success rate, basis for granting) for Discovery Grant appeals submitted to NSERC over the past few years. As far as I can tell, NSERC does not provide this data. ## NSERC rethink: Engage Grants illustrates mission drift The Globe and Mail recently posted a story entitled “Building partnerships between businesses and universities” which highlights NSERC’s Engage Grants Program. Following the article, there appears an attribution I don’t usually see in The Globe: Content in this section is provided in partnership with the Business Development Bank of Canada. BDC provides entrepreneurs with financing, venture capital and consulting services. To find out more go to BDC.ca. I found this a bit strange so I followed the link and found the mission statement of BDC.ca: Our mission Help create and develop Canadian businesses through financing, venture capital and consulting services, with a focus on small and medium-sized enterprises (SMEs). Our vision Accelerate entrepreneurs’ success. In fact, we’re solely dedicated to Canadian entrepreneurs. We have a nationwide team helping more than 29,000 businesses reach their full potential. The Engage program is now three years old. NSERC President Suzanne Fortier has characterized the program as providing funds for academic-industrial “first dates”. The Globe article contains an anecdote on one (reportedly among 240 to date [Correction: there are now over 1000 Engage grants.]) academic researcher who is having fun “making lane changes”. NSERC Vice President for Partnership Programs Janet Walden’s assertion that the program is “win-win” for business and academic sectors is not justified with any statistics or summary facts about the program as a whole. Not everyone sees it this way but the businesses helped by BDC probably like the Engage program a lot. Who wins? NSERC President Suzanne Fortier Consider the details of the Engage program: • NSERC provides$25K of taxpayer funds to pay for a six-month research and development project between a university researcher and a company already involved in research and development.
• The company is not required to invest any money on the project.
• Any intellectual property developed by the project is owned by the company. There is no direct return back to taxpayers on their investment.
• NSERC has not revealed conversion rates of Engage grants into Collaborative Research and Development program grants. Why?

Meanwhile, 240 1000 Engage grants, each costing $25K, bleed 6 25 Million dollars away from other NSERC programs like the Discovery Grants which support basic research by Canada Research Chairs and other university researchers. NSERC’s foray into business development should be contrasted with the Council’s original mission: “…encourage excellence in research; provide a base of advanced knowledge in the universities; assist in the selective concentration of research activities; aim for a regional balance in scientific capability; maintain a basic capacity for research training; encourage curiosity-oriented research; and encourage research with a potential contribution to national objectives. … these objectives are intended … to ensure long-term coherence in the federal system of university research granting.” (Honourable Hugh Faulkner, then Minister of State for Science and Technology, during the opening comments of the second reading of Bill C-26.) Panel Chair Tom Jenkins, Minister of State (Science and Technology) Gary Goodyear, Arvind Gupta, Monique F. Leroux and Nobina Robinson (Not shown are panel members David Naylor and Bev Dahlby) With Engage and other programs aimed at building academic-industrial partnerships, the current leadership has drifted away from NSERC’s core mission. This “mission drift” at NSERC was highlighted (see page 128 in the PDF version or follow this link) in the Review of Federal Support to Research and Development produced by the federal R&D panel chaired by Tom Jenkins (who was recently appointed an Officer of the Order of Canada): The granting councils have played a pivotal role in developing both talent and ideas for Canada’s innovation agenda. Their core raison d’être has been and remains investigator-initiated research of both a basic and applied nature, and each needs to continue to be generously supported. However, there has been mission drift for the granting councils, as they have responded to pressure from government to be more business facing. By excogitating the recommendations of the expert panel report, Minister Goodyear and Prime Minister Harper are poised to lead Canada’s research and development enterprise back to the launch pad: basic research; dream big. ## 2011 Winter CMS Town Hall Notes The 2011 Winter CMS Meeting took place this past weekend. On Sunday (2011-12-11), there was a CMS Town Hall meeting. The discussion was led by a panel consisting of Long Range Plan Committee Chair Nancy Reid, CMS President Jacques Hurtubise, and CMS Director Johan Rudnick (who was partially obscured by a poinsetta). Attendees to the town hall meeting were fed a free lunch. ### Long Range Plan Update Nancy gave a status update on the long range plan. She reported that the committee met in October and is in the process of writing. They hope to have a draft version of the report available in late February with a target goal of final publication in June 2012. Nancy also reported that she and Math-NSERC Liaison Committee Chair Walter Craig had sent a letter of recommendations to NSERC for this year’s Discovery Grants competition. The letter is posted here and also on the LRP web space. Nancy reported that the overall federal funding envelope for math/stats through NSERC (Discovery grants, Institute, …) is around \$21M/y. To function effectively, the Institutes require additional funds from the Provinces and other partners. To avoid the departures of talented mathematicians from Canada as forecasted last year, there is a need for more funding. I asked if the LRP report would contrast the circumstances faced by Canada’s financially threatened math/stats community with the recent \$100M ( \$50M federal, \$50M Ontario) gift to the Perimeter Institute. Nancy replied that the LRP report will only address funds granted through NSERC and will not comment on grants given through other sources. ### Persistent Concerns about NSERC and Discovery Grants The open discussion revealed that there remain serious concerns within the Canadian mathematical community about NSERC and the Discovery Grants program. Nancy reported that consultations with NSERC have “not always been easy.” Brett Stevens expressed the view that the weight given to the training of highly qualified personnel in evaluating the merit of proposals was problematic. Nancy relayed that these issues had been raised with Isabelle Blain. Based on those conversations, Nancy predicted that no substantial review of the new system would take place until 2014. According to Nancy, NSERC staff claims that no discipline other than mathematics has complained about the new peer review system. I relayed that there have been reports by computer scientists and physicists of troubles with the outcomes of their recent competitions. NSERC might not be aware of the troubles in other disciplines but that doesn’t mean they don’t exist. Through the Liaison Committee, the LRP process, the Institutes, meetings of chairs, trans-Canada research collaborations, and meetings of the CMS and SSC, the math/stats community in Canada has strong communications channels. These channels allowed us to see the anomalies in the 2011 outcome through a national lens. I wonder if other disciplines would see problems with the conference model if they too had a wide enough vantage point on the outcome of their recent Discovery Grants competitions. There was some discussion about how NSERC has moved funds away from the Discovery Grants program into a potpourri of programs supporting commercialization and academic-industrial partnerships. As mentioned in the 2007 report of the International Review panel (which led to the new peer review system), pure mathematics is unfairly punished when basic research funds are redirected toward short term commercialization goals. Eddie Campbell isolated an issue that our community must confront. Should NSERC fund lots of smaller grants or fewer larger grants? There is a fixed amount of money in the federal budget for mathematics. How should those funds be spread out? Jacques mentioned that “spreading peanut butter” is a frequent metaphor used in discussions around this issue. ### Transparency I highlighted Minister Tony Clement’s call for federal government transparency. With this background, I asked Nancy if the LRP could arrange for the release of NSERC President Suzanne Fortier’s slides from her public presentation at the Summer 2011 CMS meeting. Nancy replied that the LRP has the slides but has been instructed not to circulate them. Nancy also reported that the LRP had requested data regarding the appeals numbers and success rate. She said the success rate is running around 25% but the LRP had not yet received the requested data. In light of Mr. Clement’s call for federal government transparency, I wonder why the data presented publicly by NSERC’s President to the Canadian Mathematical Society and requested by the LRP remains concealed from public view. ### Immigration Policy Concerns David Pike expressed frustration that Canada’s immigration rules prevent international graduate students from seeking permanent residency. I didn’t quite understand the details. David reported that the policy is seriously affecting his finishing PhD student who wishes to stay in Canada but will probably be forced to leave. Canada and the provinces invest heavily in the training of international graduate students. David reported that current immigration policy prevents Canada and the provinces from benefiting from this investment since these highly qualified graduates are often required to leave after earning their PhD. The CMS town hall meeting was a great opportunity for discussion among members of the Canadian math/stats community. I look forward to the LRP report and am grateful for all the hard work that Nancy Reid and the committee have done. ## NSERC Peer Review System is Broken for Mathematics Anomalous results of the 2011 NSERC Discovery Grants competition in mathematics have provoked a loss of confidence in the NSERC peer review system. To avoid a substantial loss of Canada’s scientific talent, which has been enhanced through the Canada Research Chairs program and other spectacular hiring over the past ten years, scientific policymakers need to quickly fix the broken peer review system. In the absence of an effective peer review process setting the strategy for research investment, Canada will miss out on the rewards made over the past decade’s recruitment of scientific talent. What is happening in other sciences? Anecdotal reports from the following sources suggest the anomalies are not restricted to the math department at Toronto: • Toronto: MATH, CHM, EEB, PHY, STA, Engineering • UBC: CS, MATH • Queens: MATH What happened to other disciplines? I’d like to know but NSERC won’t reveal the 2011 data until after the federal election. I’d like to hear from other scientific disciplines about their confidence in the recently transformed NSERC peer review system. In 2007, NSERC commissioned a review by an international committee culminating in this report. (Please find my annotated version here and a marked up version of the NSERC Management response to the 2007 International Review Committee Report.) This report made recommendations leading to fundamental changes in the peer review process for all disciplines starting in 2009. The implementation of these changes (involving the so-called conference model and binning system) and other forces have provoked a loss of confidence in the peer review process at NSERC among mathematicians at Toronto, and across Canada. ## Toronto Math Results are Anomalous The results (names omitted) of the 2011 NSERC Discovery Grants Competition for the Department of Mathematics at the University of Toronto are anomalous: • Professor A. \$29k/y to \$18k/y • Professor B. 40 to 15 • Professor C. 42 to 30 to 42 to 18 • Professor D. 26 to 18 • Professor E. 40 to 40 • Professor F. 38 to 47 • Professor G. 0 to 0 • Professor H. 15 to 13 (The numbers represent annual research grant amount in dollars for the past 5 years and the new number for the next 5 years. For a description about how mathematician’s use these funds, go here.) ### About Professor C. Consider the case of CMS award winning Professor C. In 2010, this researcher’s grant was cut from 42 down to 30. After an appeal, the grant was reinstated for one year back to 42. In this year’s competition, one year after the appeal, NSERC drops it to 18, a 57% cut. Meanwhile, Professor C’s frequent collaborator (each had more than 50% overlap of their research with the other during 2006-2011), Professor K., received 45 staying at 100% of the previous level in this competition. Will the real opinion of NSERC on Professor C’s research please stand up? Professor C’s story is quite similar to Don Fraser’s personal account. (Within the conference model, I understand that a different group of only 5 experts may have reviewed the proposals of Professors C and K. This remark can account for the inconsistency but reveals aspects of larger issues that need to be fixed.) ### About Professor B. Imagine running a successful research operation (success, former students get awards, 13 major pubs in 2006-2011) for the past five years like Professor B using 45K/y. Students are in the pipeline; postdoc candidates have been scouted; Professor B has new ideas. NSERC rewards this person with a drop from 45 down to 18, a 60% cut. This researcher is confused with the outcome: “What did I change? What should I have done differently?” ### About Professor G. This person is a (perhaps the) world leading expert on a substantial research area. It seems this person, despite spectacular research success, is unworthy of a Discovery Grant because they don’t produce enough students. It is as though Canada has a sports car and they don’t put tires on it. ## Secondary Effects Scenarios Mathematicians, of international calibre, with a steady research production stream and surrounded by young researchers have had their grants slashed by nearly 60% during the 2011 NSERC Discovery Grants Competition. • Now, imagine you are an assistant professor in Canada. You might have nice support right now, like a Sloan or an ERA. You are building a research group, spending money on HQP, scouting talent. But your funding has a finite time horizon and the Discovery Grants look unstable, unpredictable. So, would you leave Canada if you could? Fix NSERC or the young talent will leave Canada. • Now, imagine you are a recently recruited Canada Research Chair: if you just concluded that your junior faculty member might be wise to leave Canada, how do you see your department in 10 years? Would you leave Canada if you could? Fix NSERC or the CRCs will leave Canada. The system is broken and needs to be fixed. There are (at least) two main problems: • Math in Canada is treated unfairly compared to other disciplines • The Peer Review system is broken ## Math in Canada is treated unfairly The main problem with mathematics funding in Canada is the amount invested is too low. I’ve written about this before. Consider the data from 2009 of NSERC Discovery Grants (2010 is similar, 2011 is not available) over the disciplines: The average math and stats grant is$20K/y while the average over all disciplines is $41K/y. Why is it that the average scientist in Canada can expect more than double the amount a Canadian mathematician can expect? Keep in mind that Discovery Grants are primarily used to fund research personnel not expensive labs. David Wehlau’s data (posted and discussed here) reveals the trend: over the past twenty years, mathematics investment as a percentage of the total amount in Discovery Grants funding has declined from nearly 4% down to 2%. Math has received less and less funding compared to other disciplines. This subtle reallocation needs to be abruptly reversed. The unfairness then multiplies. Consider the following snippet from the 2007 international review report: Other disciplines are benefitting more from other industrially targeted NSERC programs and other sources compared to pure mathematicians. NSERC views Discovery Grants as grant-in-aid: a precursor grant leading to other sources of funds. The international review committee reports that is not the case for mathematics AND mathematicians receive on average less than half the funds received by other scientists. This is an implicit funding reallocation away from mathematics toward other disciplines and is unfair. ## Broken Peer Review System The outcome of the 2011 competition, and consistent reports (like Don Fraser’s) from the past two years, have provoked a loss of confidence in the peer review system at NSERC. To rebuild trust and avoid the departure of talented scientists, the mathematics community of Canada needs to know what happened in 2011. We need to understand why the peer review system produced the 2011 funding allocations. There has been considerable chatter in the mathematics community about the 2011 competition. However, we need people with official roles to speak officially at this time. In addition to the forthcoming data from NSERC, I hope that Section 1508, the Mathematics and Statistics Evaluation Committee will explain the 2011 evaluation process and actively participate in discussions leading to an improved system that regains the confidence of mathematicians and statisticians working in Canada. What happened? How can we fix it? ## Mathematics Discovery Grants are Insufficient and Broken Beginning this academic year, I have served the Department of Mathematics as the Associate Chair [Research]. The principal responsibility of this position is to administer the hiring process, for tenure stream and postdoctoral appointments. I also recently served on the Connaught Physical Science Review Panel which reviews applications for Connaught funds and adjudicates the McLean Award. Serving on the Connaught Panel has given me a perspective on the research climate in fields outside of mathematics. Metaphorically, these responsibilities have required me to look up from my research papers and out the window at the infrastructure supporting researchers at UofT. I am troubled by what I have seen: Here is a table generated by NSERC showing the Discovery Grants (DG) data across disciplines. # Discovery Grants are Insufficient On average, Canadian mathematicians receive about$20K/y to sustain their research program. This post by Izabella Laba explains how mathematicians typically use this money at UBC. As part of the planning exercise, I suggest we consider how much money a mathematician needs to run a research program. Simple considerations show that $20k is insufficient. • At Toronto, faculty are encouraged to contribute one quarter of their NSERC DG to support graduate students. This amounts to$6K from a typical $20K NSERC DG. • Faculty pay some incidental expenses related to computing infrastructure costs, printing, etc. Let’s suppose these total about$1K.
• Collaborative mathematical research requires travel to bring the coworkers together. A typical research visit requires an airfare purchase and a hotel room. Each research visit costs between \$1K and \$2K.
• The development of graduate students and postdocs into researchers often requires conference participation. The associated travel costs are frequently paid using the advisor’s Discovery Grant. Local expenses for sponsored graduate students are often paid by the conference. Each field trip by a graduate student or postdoc costs about $500, sometimes more if the conference is outside of North America. What does minimal research activity look like? What does it cost? Minimal research activity might involve, say, three to four research visits over the calendar year, and one graduate student research field trip. These might add up to$4K.

So, a minimally active researcher with a typical research grant of  \$20K spends \$6K supporting a graduate student, \$1K on incidentals, and \$4K on research visits leaving $9K. • At Toronto, postdoctoral positions require between \$32K and \$40K from faculty research grants. This money is supplemented with a teaching stipend to complete the salary package for the postdoc. The minimally active mathematician with average funding can barely afford one third of a postdoc. Therefore, it is necessary to combine funds with like-minded colleagues to generate a postdoc position. The need to build funding alliances prevents typical faculty members from choosing the postdoctoral candidate with the most synergy with their research program. Instead, the typical researcher needs to look for a candidate that two of their colleagues will also like enough to spend \$10K or more to have in the department. Instead of recruiting postdocs with potentially explosive overlapping research interests, we make deals just to get someone in the department and hope for some resonance after training the Postdoc on our research topic. Of course, all three faculty members want the person they hire to contribute to their research program. So the Postdoc is encouraged to learn background materials in three (hopefully related) areas which are not strongly linked to their thesis area. Except in rare cases with unexpected synergy, this arrangement is a failure factory.

Mid-Career Funding Gap

Early research awards (ERA, Sloan, etc.) fund research activities by new faculty. These opportunities are restricted by a time horizon typically around 10 years after the PhD. Young faculty at Toronto with these grants can often solely fund the research component of a Postdoc funding package.

Similarly, eminent senior members of our department with large grants can solely fund postdocs.

Midcareer mathematicians typically must make a deal with their colleagues to assemble the funds to to hire a Postdoc.

Canada’s Discovery Grants funding policies do not adequately support research activity by mathematicians and especially hurts researchers in the middle of their career. Young researchers who win early research awards in Canada are capable of starting a research program. The present funding structure does not allow these emerging mathematicians to fully develop through the mid-Career phase into world class research leaders. Canada has been very effective at recruiting talented young mathematicians during the past decade. When young research stars in Canada start to realize the mid-Career funding gap prevents them from carrying out their research plans, they will leave and go elsewhere.

## HQP training is not necessary for spectacular research success.

Why does NSERC require an HQP (Highly Qualified Personnel) component in research plans? I expect the answer has to do with the general goal that the government policies should encourage the training of a highly skilled population. This is a good goal. However, the HQP requirements for Discovery Grants fail to envision the secondary effects of world leading research on the training of future scientists. Consider, for example, the case of Jean Bourgain. Bourgain had one Ph.D student so his HQP production would be viewed as insufficient to merit a Discovery Grant by NSERC. However, Bourgain’s advances have created new fields of mathematics where a generation of mathematicians (e.g. Izabella LabaWilhelm Schlag, Gigliola Staffilani, Terry Tao …) has  blossomed. The NSERC definition of HQP fails to envision the effects of ground breaking research in the development of future scientists.

Henri Poincaré had five graduate students. John von Neumann only had three students. I wonder if they’d qualify for a Discovery Grant given the highly qualified personnel  requirements in the funding formulae? My colleague Victor Ivrii made an historical advance by proving Weyl’s conjecture about the eigenvalues of the Laplacian on a domain. Due to the HQP requirement, Victors’s DG is zero. Similarly, my colleague Michael Goldstein advances the frontier with big results which appear in Annals, Annals (again), Acta, etc. and his efforts are rewarded with a \\$14K/y NSERC DG. Goldstein’s research excellence has been recognized outside of Ottawa.

In my opinion, the fact that Canada fails to invest in the scholarly activity of its world-class researchers is a disgrace. The Discovery Grants system needs to be fixed.

## Research Program Profiles  should be Defined

The mathematical community, perhaps through the long range planning exercise, should formulate typical budgets required for mathematician’s operating at different levels. How much money is required to effectively run the research enterprise for a mathematician? Of course, the answer depends upon career stage, level of engagement with students and postdocs, etc. The mathematical community should define funding requirements for a collection of research program profiles (parametrized by quantity and quality, and HQP level ranging through zero, some, to lots of junior collaborator participation in the research). If NSERC wants to put us in bins, mathematicians should define those bins, not NSERC staff, and we should insist on adequate funding to allow Canada’s mathematical research community to emerge as world leading. Presently, there are too many government generated obstructions preventing our ascension.