From Template to Production: Building a Chat App with Azure AI Foundry

· Ofek Ben Eliezer
Azure Web App AI architecture

Build a secure, repeatable, and cost-aware chat experience over your data using the official Microsoft template, Azure App Service, and Azure AI Foundry with Prompt Flow.

This guide shows how I took Microsoft’s sample-app-aoai-chatGPT template and turned it into a production-ready deployment on Azure App Service, backed by a model deployed in Azure AI Foundry, plus Prompt Flow for evaluation and iteration.

What You Will Build

Important: the reference template uses some preview APIs. Validate feature availability in your region and subscription before you roll out to production.

Azure Web App AI architecture

Why App Service + Azure AI Foundry

Architecture

Browser
  |
  v
Azure App Service (Web App)
  |   ^
  |   | Managed Identity
  v   |
Azure Key Vault  ----> secrets / endpoints
  |
  | VNet + Private DNS
  v
Azure AI Foundry (Model endpoint)
  |
  v
Data sources (Search, Storage, SQL) via private endpoints

Prerequisites

Official resources

Links appear in the References section at the end.

Step 1 - Start from the Microsoft template

  1. Fork the template to your GitHub account.
  2. Read the README and run locally to confirm the basic chat works.
  3. Commit a minimal settings file so the app reads configuration from environment variables, not hardcoded values.

Tip: keep your fork clean. Use branches for changes, and document what you changed versus the upstream template.

  • Official repo: github.com/microsoft/sample-app-aoai-chatGPT
    • Azure Web App AI architecture

    Step 2 - Deploy a model with Azure AI Foundry

    1. In Azure AI Foundry, create or open a Project.
    2. Pick a model from the Models catalog and create a deployment. Start with a standard deployment type that matches your region and quota.
    3. Note the endpoint, deployment name, and the API version shown in the portal.
    4. If you are using On Your Data, connect your data source and complete access configuration.

    Security tip: prefer private endpoints for the model resource and your data plane. This allows the app to call the model over a private link rather than the public internet.

    Azure Web App AI architecture

    Step 3 - Host the app on Azure App Service

    You have two common paths: deploy straight from GitHub Actions, or script the infrastructure first and then publish.

    Option A - GitHub Actions CI/CD

    Example workflow snippet:

    name: deploy-appservice
on:
  push:
    branches: [ main ]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: Build
        run: |
          # install deps and build
          echo "Build your app here"
      - uses: azure/webapps-deploy@v3
        with:
          app-name: ${{ vars.WEBAPP_NAME }}
          package: .
    Azure Web App AI architecture

    Option B - Script the infrastructure first

    For repeatability, create the App Service plan and Web App, enable System Assigned Managed Identity, and use Key Vault references for your secrets. You can do this with Bicep, Terraform, or the Azure CLI.

    Minimal CLI example:

    # Resource group
az group create -n rg-chat-on-appservice -l westeurope

# App Service plan
az appservice plan create \
  -g rg-chat-on-appservice -n plan-chat --is-linux --sku P1v3

# Web app with managed identity
az webapp create \
  -g rg-chat-on-appservice -p plan-chat -n web-chat-on-appservice \
  --runtime "PYTHON:3.12" \
  --assign-identity

    Step 4 - Store configuration in Key Vault and bind with MI

    1. Create a Key Vault and add secrets like the model endpoint or connection strings.
    2. Grant the Web App’s managed identity get and list permissions to secrets.
    3. In App Service Configuration, reference Key Vault values using the @Microsoft.KeyVault(SecretUri=...) syntax.

    Example app settings:

    OPENAI_ENDPOINT = @Microsoft.KeyVault(SecretUri=https://kv-chat.vault.azure.net/secrets/openai-endpoint/...)
OPENAI_DEPLOYMENT = gpt-4o-mini
OPENAI_API_VERSION = 2024-08-01-preview

    Note: when you use a user assigned identity, set keyVaultReferenceIdentity on the Web App to that identity. For system assigned identity, the platform uses it automatically.

    Step 5 - Network hardening

    This setup keeps model calls and data access inside your network path.

    Step 6 - Plug in Prompt Flow for quality

    Use Prompt Flow to version your prompts, run batch evaluations, and compare runs. The flow runs against your deployed model endpoint and produces metrics like accuracy proxies, groundedness, and cost.

    1. Create a flow for your main system prompt with inputs and few-shot examples.
    2. Prepare a small evaluation dataset that reflects your domain.
    3. Run an evaluation and inspect outputs and cost.
    4. Iterate on the prompt or tool calls, then re-run.

    Document your best run in the repo so others can replicate it.

    Prompt Flow
    Example of Prompt Flow in action, showcasing its capabilities for managing and evaluating prompts.

    Observability

    Cost and sizing

    Common pitfalls

    Message
    Example of ChatGPT model response in the web app.

    References

    Latest Video - Showing this Use Case From Microsoft Reactor (Hebrew)

    Next steps

    If you use this approach in production, consider sharing a short write-up with your architecture diagram and any lessons learned. It helps the community move faster.