[Azure AI Content Safety - Azure AI Content Safety] API Review

[azure-sdk]

New API Review meeting has been requested.

Service Name: Azure AI Content Safety - Azure AI Content Safety
Review Created By: Meng Ai
Review Date: 10/26/2023 04:00 PM PT
Onboarding Record: https://dev.azure.com/azure-sdk/Release/_workitems/edit/17868
PR:
Hero Scenarios Link: Not Provided
Core Concepts Doc Link: Not Provided

Description:

Detailed meeting information and documents provided can be accessed here
For more information that will help prepare you for this review, the requirements, and office hours, visit the documentation here

[azure-sdk]

Meeting updated by Meng Ai

Service Name: Azure Content Moderator - Azure AI Content Safety
Review Created By: Meng Ai
Review Date: 10/26/2023 04:00 PM PT
Onboarding Record: https://dev.azure.com/azure-sdk/Release/_workitems/edit/17868
PR: #25570
Hero Scenarios Link: Not Provided
Core Concepts Doc Link: Not Provided

Description: We've added two new APIs to do Jailbreak detecting and nl snippy detecting in 2023-10-15-preview version.
And if there is enough time, let's go through 2023-10-30-preview contract: #26317

Detailed meeting information and documents provided can be accessed here
For more information that will help prepare you for this review, the requirements, and office hours, visit the documentation here

[mikekistler]

Notes from API Review meeting 10/26/23

• Question on use case of protected material detection -- why flag these as inputs?
  • The intent is to use this on LLM responses
• Is "JailBreak" better that "PromptInjection" ?
  • Marketing recommended "JailBreak"
  • "JailBreak" is a superset of "PromptInjection"
• Do you want to let users choose specific checks to perform?
  • It might get expensive (COGS) to perform all checks as you add more forms of checks
• Is it possible to tell people what the violation is? Users will probably want an explanation
• Make the response structures consistent between these two APIs -- wrap or not in an "analysis" object

Notes on review of #26317

• Path should be "/path/incidents" -- plural "incidents" since this is a collection
• Consider shortening path segment "incidentSamples" to just "samples"
• incidentSamples should really be a resource and follow our standard resource patterns
  • You can still support a batch upload
  • Recommend studying this in your preview
• Consider merging IncidentAnalysis and IncidentMatches into a single object

These all look good for preview. Please consider the above points and either make changes or use your preview to evaluate if changes are needed.

[mengaims]

Notes from API Review meeting 10/26/23

• Question on use case of protected material detection -- why flag these as inputs?

  • The intent is to use this on LLM responses

• Is "JailBreak" better that "PromptInjection" ?

  • Marketing recommended "JailBreak"
  • "JailBreak" is a superset of "PromptInjection"

• Do you want to let users choose specific checks to perform?

  • It might get expensive (COGS) to perform all checks as you add more forms of checks

• Is it possible to tell people what the violation is? Users will probably want an explanation

• Make the response structures consistent between these two APIs -- wrap or not in an "analysis" object

Notes on review of #26317

• Path should be "/path/incidents" -- plural "incidents" since this is a collection

• Consider shortening path segment "incidentSamples" to just "samples"

• incidentSamples should really be a resource and follow our standard resource patterns

  • You can still support a batch upload
  • Recommend studying this in your preview

• Consider merging IncidentAnalysis and IncidentMatches into a single object

These all look good for preview. Please consider the above points and either make changes or use your preview to evaluate if changes are needed.

@mikekistler The comments for #25570 has been resolved:
"Is it possible to tell people what the violation is? Users will probably want an explanation"

• We will add explanation for analyze text API in 2023-12-30-preview
  “Make the response structures consistent between these two APIs -- wrap or not in an "analysis" object”
• The response structures has been aligned:
  Jailbreak:
  {
  "jailbreakAnalysis": {
  "detected": true
  }
  }

ProtectedMaterials:
{
"protectedMaterialAnalysis": {
"detected": true
}
}