AI in Canada AI in Canada

Privacy considerations when using AI

Things to know

• Canadian privacy laws apply if personal information is used to fine-tune an AI model, if a prompt entered into an AI model or system includes personal information, or if an AI model or system is used to generate output that includes personal information.

• “Personal information” is broadly defined under Canadian privacy law and includes information that can be used alone or in combination with other information to identify an individual.
• Unlike under privacy laws in some jurisdictions, such as the European Union, Canadian privacy laws are consent-based with limited exceptions to consent and do not include a “legitimate interests” legal basis for processing personal information.
• Compliance obligations vary depending on the nature of the organization and the industry in which it operates (e.g., financial services, telecommunications, retail, health or public sector), the nature and sensitivity of the personal information that is processed, and the activities the organization undertakes using an AI model or system (e.g., automated decision-making to grant a loan or select a job applicant).

Things to do

• Identify and document your legal authority (i.e., consent or an exception to consent) for collecting and using personal information to fine-tune an AI model or to prompt an AI model or system to generate output. When relying on consent, ensure that the consent is valid and meaningful. When sourcing personal information from a third party, obtain assurances that the information was collected lawfully and the third party has authority to disclose it to you for the intended purposes.
• Remember that publicly available personal information, including information published online, is subject to privacy laws in Canada (including when subject to an exception to consent).
• Be open and transparent (e.g., in your publicly-posted privacy policy and user flows) with individuals about what, how, when and why personal information is collected, used or disclosed during fine-tuning or use of an AI model or system, and provide this information in an understandable manner. Disclose any known limitations about the accuracy of AI outputs (e.g., age of data used to train the model) and any known or likely risks.
• Collect, use and disclose personal information only for documented, legitimate and appropriate purposes.
  • Avoid inappropriate uses of AI models or systems, including uses that violate “no-go zones” identified by the Office of the Privacy Commissioner of Canada, such as “profiling or categorization that may lead to unfair, unethical or discriminatory treatment that is contrary to human rights law; the collection, use, or disclosure of personal information for purposes that are known or likely to cause significant harm to individuals or groups, or activities which are known or likely to threaten fundamental rights and freedoms.”

• Consider whether the use of an AI model or system that involves the collection, use or disclosure of personal information is necessary and proportionate and whether there are more privacy protective technologies that can be used to achieve the same purpose.
• Avoid prompting an AI model or system to re-identify data that has been previously de-identified.
• Consider whether the output of an AI model or system is accurate and reliable in the context of the intended purpose.
• Collect, use, retain and disclose personal information only to the extent needed to fulfill the explicitly specified, appropriate purpose.
  • Do not use personal information when using anonymized or synthetic data will work for your purposes.
  • Where personal information is required, remove direct identifiers (e.g., from the prompts or output) wherever possible, such that only de-identified personal information remains.

• Ensure that personal information used to fine-tune an AI model is accurate as necessary for the purpose of the model.
• Perform (and update over time) privacy impact assessments, algorithmic impact assessments and bias testing and assess resilience to inferencing or other attacks.
• Develop a comprehensive AI governance program and/or review and enhance your privacy governance program and security policies and practices to address AI-related issues. This includes enabling individuals to request access to their personal information, ask questions, submit complaints and correct inaccurate personal information.
• For public-facing AI tools, ensure individuals know they are interacting with an AI tool and inform individuals of the privacy risks and options available to them.
• When using an AI system as part of a decision-making process:
  • clearly communicate to affected individuals the use of the system to make a decision, the general functioning of the system and how the system is used
  • clearly communicate, or be ready to communicate, to affected individuals how a decision that may have a significant impact on them was reached (including the personal information that was used to reach that decision), how to request a human review or reconsideration of the decision, how to request a correction of the personal information used, and any other recourse options
  • where appropriate, including where a decision will have a significant impact on the individual, include a human reviewer in the decision-making process
  • maintain adequate records to allow for requests by an affected individual for access to information about a decision
  • ensure decisions that relate to a specific group are made only after determining that the group is adequately and accurately represented in the system’s training data
• Consider if, or to what extent, laws of general application (such as human rights laws and employment laws) apply to use of your AI model or system.
• Consider if, or to what extent, laws of other jurisdictions (such as the E.U.’s Artificial Intelligence Act or General Data Protection Regulation) apply to the use of your AI model or system.

Useful resources

• “Principles for responsible, trustworthy and privacy-protective generative AI technologies,” Joint publication of the Office of the Privacy Commissioner of Canada and provincial data privacy regulators, December 7, 2023

---

• Human rights