Using Copilot Studio as MCP Client Connecting to Your Custom MCP Server deployed locally or Azure Container Apps

DevOps - CLU Agent

===========================================================================

Introduction

Integrating Copilot Studio with a custom Model Context Protocol (MCP) server opens up powerful possibilities for extending your AI assistant's capabilities. Once you've set up your MCP server and exposed it via DevTunnel (accessible at https://5ldvknx5-8000.use.devtunnels.ms/mcp), you can configure Copilot Studio to act as an MCP client. This connection allows Copilot Studio to leverage your custom tools and data sources, such as querying pull requests, branches, and commits from your repositories. The MCP protocol provides a standardized way for AI assistants to communicate with external services, making it easier to build context-aware applications without hardcoding integrations.

Pre-requisites

Clone the GitHub repository and check the README.md for setting up the MCP Server locally:

GitHub Repo: https://github.com/Anilalkg/MCP-Server.git

Get the streamable HTTP URL as detailed in the setup (e.g., https://5ldvknx5-8000.use.devtunnels.ms/mcp)
Recommended Code Editors: VS Code or Anti-Gravity

Setup Steps

Step 1: Navigate to Copilot Studio (CPS)

Open Copilot Studio in your browser and log in to your account.

Step 2: Create a Blank Agent

Click on Create or New Agent
Select Blank Agent to start from scratch

Step 3: Add a Tool

Navigate to the Tools section
Click Add a tool
Select Model Context Protocol

Step 4: Configure the MCP Connection

Fill in the following details:

Server name: (e.g., "Agent DevOps MCP Server")
Server description: (e.g., "Custom MCP server for Azure DevOps operations")
Server URL: Enter your streaming URL - https://5ldvknx5-8000.use.devtunnels.ms/mcp

Step 5: Create and Connect

Click Create Connection
Select Connect to Agent DevOps
Click Add and Configure

Now the tools should be added successfully!

Step 6: Verify Tool Integration

In your Tools home page, under the Tools menu, you should see all the methods decorated with @mcp.tool() from your MCP server configurations.

Screen should look like this




Testing Your Agent

Now you should be able to ask questions in your Copilot Studio Test Panel. Here are some sample questions this agent can answer:

>list me all the pull requests
>list me all the branches available in AI-Solutions
>list me all the user names of the persons who did the commits to AI-Solutions

Conclusion

This architecture provides significant flexibility for enterprise scenarios where you need to connect AI assistants to internal systems, databases, or specialized APIs. By using the MCP protocol as a bridge, you maintain a clean separation between your AI interface (Copilot Studio) and your backend services. This means you can update your MCP server's functionality independently without reconfiguring Copilot Studio, and you can even reuse the same MCP server across multiple AI platforms.

Whether you're building developer tools, IT support bots, or business intelligence assistants, the combination of Copilot Studio and custom MCP servers creates a robust foundation for AI-powered workflows.

================================================================================

Tags: #CopilotStudio #MCP #AzureDevOps #AI #Automation #ModelContextProtocol

Microsoft Copilot Studio: How to Generate Dynamic Adaptive Cards Contents

 Introduction

Working with Microsoft Copilot Studio often requires creating dynamic, interactive user experiences. One powerful feature that enables this is the integration of Adaptive Cards with dynamic JSON content generation. In this post, I'll walk you through a real-world implementation where we needed to generate radio buttons dynamically based on items returned from a RAG (Retrieval-Augmented Generation) system.

The Challenge

Our use case involved creating a conversational interface where users could select from a list of options that weren't static. The radio button choices needed to be generated dynamically based on the results returned from our RAG implementation. This meant we couldn't rely on pre-built, static Adaptive Cards.

The Solution: Dynamic Variable Integration

To solve this challenge, we leveraged Microsoft Copilot Studio's Dynamic Variable functionality combined with Adaptive Card Templates. Here's how we implemented it:

Step 1: Setting Up the Adaptive Card Template

In Copilot Studio, we configured an Adaptive Card Template using the `AdaptiveCardTemplate` object with placeholder expressions:

```yaml
- kind: SendActivity
  id: sendActivity_dinNxr
  displayName: Dynamic Radio Button
  activity:
    text:
      - "{Text(Topic.KBAresponse.answer)}"
    attachments:
      - kind: AdaptiveCardTemplate
        cardContent: =Text(Topic.Var1RadioButton)
```

The key here is the `cardContent: =Text(Topic.Var1RadioButton)` line, which references our dynamic variable that contains the JSON structure for our Adaptive Card.

Step 2: API Integration for Dynamic Content

The dynamic variable `Topic.Var1RadioButton` gets populated through an HTTP Node in Copilot Studio, which makes an API call to a Python-hosted FastAPI endpoint. This approach allows us to generate the Adaptive Card structure programmatically based on the current context and data.

Step 3: FastAPI Implementation

Here's the Python FastAPI implementation that generates our dynamic Adaptive Card content:

```python
from fastapi import APIRouter, Request
from fastapi.responses import PlainTextResponse
import json

router = APIRouter()

@router.post("/generate-adaptive-card")
async def generate_adaptive_card(request: Request):
    payload = await request.json()
    solution_ids_used = payload.get("SolutionIDsUsed", [])
    used_ids = payload.get("UsedIDs", [])
    
    # Generate your adaptive card JSON structure here
    # adaptive_card = {...}  # Your card logic
    
    # Return escaped string (for plain text transport of JSON)
    escaped_json = json.dumps(adaptive_card)
    return PlainTextResponse(content=escaped_json)
```

Critical Implementation Details

Return Type Matters

One crucial aspect of this implementation is the return type from your API endpoint. The response **must be of type Plain Text**, not JSON. If you return a JSON response directly, Copilot Studio's workflow screen will throw rendering errors.

This is why we use:
```python
return PlainTextResponse(content=escaped_json)
```

Instead of returning the JSON object directly.

Dynamic Variable Population

The HTTP Node in Copilot Studio should be configured to:
1. Make a POST request to your FastAPI endpoint
2. Send the necessary context data (like `SolutionIDsUsed` and `UsedIDs`)
3. Store the response in your dynamic variable (`Topic.Var1RadioButton`)

Conclusion

Dynamic Adaptive Cards in Microsoft Copilot Studio open up powerful possibilities for creating responsive, data-driven conversational experiences. By combining Dynamic Variables with external API endpoints, you can create truly dynamic user interfaces that adapt to your users' needs and context.

An error has occurred. Error code: HttpRequestFailure Conversation Id: e3QGxxxxxx01NA-us Time (UTC): 2025-06-01T16:14:05.372Z.

 

Troubleshooting MS Copilot Studio HTTP Request Failures: A Practical Solution

The Problem Description

If you're working with MS Copilot Studio, you've likely encountered this frustrating error:

MS Copilot Studio -- 

An error has occurred. Error code: HttpRequestFailure Conversation Id: e3QGxxxxxx01NA-us Time (UTC): 2025-06-01T16:14:05.372Z.

This is a common issue that occurs frequently with MS Copilot Studio, though not consistently. Based on my experience, this isn't a code issue but rather a configuration issue related to the MS environment.

What We Discovered

When we enabled the Continue on Error feature in Copilot Studio, we encountered an HTTP 408 error (Request Timeout). This means the server didn't receive a complete request from the client within the expected timeframe.

After researching and contacting our support counterpart, they provided the following stacktrace:

CorrelationId: xxxx-b674-4c2a-xxxx-184197387367
Exception: Microsoft.IdentityModel.S2S.S2SAuthenticationException: S2xx2099: 
An exception has been caught while validating the request. 
Exception: [PII of type 'System.AggregateException' is hidden]

---> System.AggregateException: S2xx2096: Microsoft.IdentityModel.S2S.JwtAuthenticationHandler 
caught exceptions when validating the token. See AuthenticationResult.InboundPolicyEvaluationResults 
for additional details. (S2xx2086: An exception has been caught while validating the request 
applying the policy with id : 'User'. 

Exception: Microsoft.IdentityModel.Tokens.SecurityTokenInvalidAudienceException: IDXxx214: 
Audience validation failed. Audiences: 'xxx-xxx-4af1-b9a8-09a648fb6699'. 
Did not match: validationParameters.ValidAudience: 'null' or validationParameters.ValidAudiences:

Unfortunately, the issue remains unresolved as of when I write this post.

Our Temporary Solution: Retry Logic

As a quick fix, we implemented retry logic in the Copilot Studio workflow UI using the Goto Step feature for HTTP nodes, with a maximum retry count of 3.

Implementation Code

Here's the complete code snippet for our temporary fix:

1. Initialize Retry Counter

- kind: SetVariable
  id: setVariable_btVmH4
  displayName: retryCount
  variable: Topic.retryCount
  value: 0

2. HTTP Request with Error Handling

- kind: HttpRequestAction
  id: JDNAtp
  displayName: PF - HTTP Request
  method: Post
  url: https://pf-end-point-autoendpoint.eastus.inference.ml.azure.com/score
  headers:
    Authorization: Bearer 6mIiASFPiZKeaRxxUk4JQQJ99BGAAAAAAAAAAAAINFRAZML2PC6
    azureml-model-deployment: auto-20250708-551705
    Content-Type: application/json

  body:
    kind: JsonRequestContent
    content: |
      ={
          question:Topic.UserQuery,
          chat_history:Global.VarHistory
      }

  errorHandling:
    kind: ContinueOnErrorBehavior
    statusCode: Topic.ErrorStatusCode

  requestTimeoutInMilliseconds: 30000
  response: Topic.KBAresponse
  responseSchema: Any
  responseHeaders: Topic.ResponseHeader

3. Error Status Logging

- kind: SendActivity
  id: sendActivity_dxnBNR
  activity: After HTTP request --- Error Code--- {Topic.ErrorStatusCode}

4. Retry Logic Implementation

- kind: ConditionGroup
  id: conditionGroup_cN6dbL
  conditions:
    - id: conditionItem_xH0b5H
      condition: =!IsBlank(Topic.ErrorStatusCode)
      displayName: Condition to try Retry Logic
      actions:
        - kind: SetVariable
          id: setVariable_9Lszte
          variable: Topic.retryCount
          value: =Topic.retryCount + 1

        - kind: ConditionGroup
          id: conditionGroup_UA3fsX
          conditions:
            - id: conditionItem_itQSEP
              condition: =Topic.retryCount < 3
              actions:
                - kind: SendActivity
                  id: sendActivity_Y0iihp
                  activity: ---->{Topic.retryCount}---

                - kind: GotoAction
                  id: S4S1LT
                  actionId: JDNAtp

        - kind: SendActivity
          id: sendActivity_vOX9b5
          activity: Errorred --{Topic.retryCount}

How It Works

1. Initialize a retry counter to 0
2. Execute the HTTP request with error handling enabled
3. Check if an error occurred (ErrorStatusCode is not blank)
4. Increment the retry counter
5. Retry up to 3 times using the GotoAction to jump back to the HTTP request
6. Log the final error state if all retries fail

Key Takeaways

• This appears to be an authentication/token validation issue on Microsoft's end
• The Continue on Error feature is essential for implementing retry logic
• Retry logic provides a practical workaround while waiting for Microsoft to resolve the underlying issue
• The GotoAction feature in Copilot Studio makes implementing retry patterns straightforward

Conclusion

While this isn't a permanent solution, it significantly improves the reliability of HTTP requests in MS Copilot Studio workflows. If you're experiencing similar issues, consider implementing this retry pattern until Microsoft addresses the root cause.

Have you encountered similar issues with MS Copilot Studio? Share your experiences and solutions in the comments below!

---

Tags: #MSCopilotStudio #Azure #HTTPErrors #RetryLogic #Troubleshooting #Microsoft

Deploy the Prompt Flow Code to Azure AI Foundry : A step by step guide

Introduction:

 
Azure AI Foundry, combined with AZ Prompt Flow, provides a robust framework for building and deploying AI-driven applications. By following this guide, you can efficiently develop AI workflows, deploy them to a managed online endpoint using either the Azure ML CLI or the Python SDK, and integrate them with various services for real-time inference.

Key Benefits of Azure AI Foundry

- Fully managed: Azure handles the infrastructure and management of your AI deployment, reducing operational overhead

- Optimized AI workflows: Streamlined processes for developing, deploying, and managing AI models, with built-in automation features

- Built-in inferencing: Native capabilities for running trained models and generating predictions from new data

- Seamless AI model integration: Easily incorporate your own models or pre-trained ones into the Foundry environment

Once your Prompt Flow is tested and validated, you can deploy it to Azure AI Foundry for real-time inferencing. 

Inferencing is the process of applying new input data to a machine learning model to generate outputs.

Deployment Approaches

As told before, there are two primary methods for deploying your Prompt Flow to Azure AI Foundry:

1. Using Azure ML CLI

2. Using Python SDK:

In this blog, I will explain the approach #1: Using Azure ML CLI.

Step-by-Step Process for Azure ML CLI Deployment

The following steps will guide you through deploying a flow as a model in Azure ML, creating an online endpoint, and configuring deployments. This assumes you have tested your flow locally and set up all necessary dependencies, including the Azure ML workspace and required connections.

Azure ML CLI Approach

Pre-requisite :

1 Install Azure CLI and ML extension

>az extension add --name ml --yes

Use the below CLI to validate AZ ML is installed correctly

>az extension show --name ml

2 Make sure you have created the connection used in the flow in your Azure ML workspace

Deployment Steps

1. Registering a Machine Learning Flow as a Model in Azure ML

This command registers a defined machine learning flow as a managed model within Azure ML. This enables versioning, deployment, and MLOps capabilities for the flow.

Define the model metadata in a model.yaml file.
 This file describes the model's name, version, and location.

> az ml model create --file honda-prod-model.yaml

Sample YAML for reference :

2. Creating an Online Endpoint for Real-time Inference

> az ml online-endpoint create --file honda-prod-endpoint.yaml

This command registers the endpoint with Azure ML and provisions the necessary infrastructure to handle incoming requests. 

The honda-prod-endpoint.yaml file contains all the configuration details for your endpoint, including the name, authentication mode, and compute specifications.

After the endpoint is successfully created, you'll receive a response with the endpoint details, including its scoring URI. You can then proceed with deploying your model to this endpoint and configuring traffic distribution to optimize performance.

Sample YAML for reference :

3. Creating an Online Deployment in Azure ML

An online deployment in Azure Machine Learning (Azure ML) is a containerized environment where your model runs and serves real-time predictions. Each deployment is associated with an online endpoint, and multiple deployments can be managed under the same endpoint to support A/B testing, versioning, or gradual rollouts.

Deploying with 0% Traffic

To create a deployment without initially routing any traffic to it, use the following command:
>az ml online-deployment create --file honda-deployment.yaml

Deploying with 100% Traffic

Once the deployment is verified and tested, you can direct all traffic to it using:

>az ml online-deployment create --file honda-deployment.yaml --all-traffic

Sample YAML for reference :

Test the Deployed Model by Invoking the Endpoint

Use the below command to test whether the deployments are working:

>az ml online-endpoint invoke --name honda-chat-endpoint --request-file sample-request.json

Here's an example of what your sample-request.json might look like:

{

  "input_data": {

    "input_string": "What are the maintenance intervals for a 2024 Honda Civic?"

  }

}

You can also test the endpoint using other tools like Postman or curl. When using these tools, you'll need:

The scoring URI (available from the endpoint details)
An authentication key or token
Properly formatted request payload

Conclusion

Using the Azure ML CLI provides a straightforward way to deploy your Prompt Flow to Azure AI Foundry. The command-line approach offers flexibility and can be easily incorporated into CI/CD pipelines for automated deployments.

Reference : https://github.com/Azure/azureml-examples

Deploying Azure ML Prompt Flow to Azure as App Service: A step by step guide

Introduction

When exploring deployment options for my recent Azure ML Prompt Flow project, I found that Azure App Service offered the perfect balance of simplicity and functionality. This approach stood out for its ability to get AI applications into production quickly with minimal overhead.

My Integration Scenario

My specific use case centered around having Microsoft Copilot Studio handle all UI orchestration, using Direct Line API to connect via BotFramework-WebChat. Microsoft Copilot Studio effectively communicates with the backend deployed Azure Prompt Flow through REST API workflow tasks. The BotFramework-WebChat implementation was architected around a Redux state management pattern, ensuring efficient data flow and a responsive, dynamic user experience.

Solution Architecture

Architecture Diagram - Azure ML Prompt Flow with App Service

Why Choose Azure App Service for Prompt Flow?

- Quick Deployment: Get your AI flows into production faster

- Minimal Infrastructure Management: Focus on your application, not infrastructure

- Perfect for Smaller to Medium-Scale Applications: Right-sized solution

- Simple Yet Scalable Solution: Start small and scale as needed

Step-by-Step Deployment Guide

Pre-requisites and Environment Setup

Before starting the deployment process, ensure you have the following prerequisites installed and configured:

- Python 3.12 or  higher

- PowerShell or Git-Bash

- Azure CLI

- Docker Desktop (installed and configured)

- Conda Virtual Environment (created and activated)

- Prompt Flow Package (installed via pip)

Local Setup: Building and Testing Azure ML Prompt Flows

To begin, I cloned the Microsoft Prompt Flow repository as the foundation for my local development environment. Building directly on this codebase allowed me to leverage the existing Prompt Flow CLI and core functionalities.

Repository: https://github.com/microsoft/promptflow.git

Within this cloned repository, I created my specific Prompt Flow in VS Code, tailoring it to my application's requirements. To ensure proper functionality before deployment, I followed these steps for local testing:

1. Connection Setup: I established the necessary connections, such as the Azure OpenAI connection, using the Prompt Flow CLI:

   ```bash

   pf connection create --file .\honda-PROD\azure_openai_connection.yaml

   ```

   This step ensured that my flow could access the required AI models and services without issues.

2. Local Flow Serving: I then served my Prompt Flow locally using the `pf flow serve` command:

   ```bash

   pf flow serve --source .\honda-PROD\ --port 8085 --host localhost

   ```

   This allowed me to access my flow via `http://localhost:8085/` for immediate testing and iteration.

Preparing for Azure App Service Deployment

Build and Deploy the FLOW Setup

Steps:

1. Login to Azure portal using CLI

   ```bash

   az login    # Authenticate yourself

2. Create Resource Group in the Azure Portal

   ```bash

   az group create --name rg-for-honda-pf-app-service --location eastus2

   az group list --output table/json  

   ```

3. Create Container Registry in the Azure Portal

   ```bash

   az acr create \

     --resource-group <resource-group-name> \

     --name <container-registry-name> \

     --sku <sku> \

     --location <region>

   ```

   Handy Sample AZ-CLI

   ```bash

   az acr create --name mycrforpf --resource-group rg-for-honda-pf-app-service --sku ASP-P0v3-1 --location eastus2

   az acr update --name mycrforpf --admin-enabled true

   ```

4. Build the FLOW as docker format app

   Use the below command to build a flow as a docker format app:

   ```bash

   pf flow build --source ../../flows/standard/web-classification --output dist --format docker

   ```

   This will generate the DockerFile for you inside the dist folder.

5. Deploy the FLOW to Azure Portal as App Service

   The code provided by Microsoft is available in:

   - `/examples/tutorials/flow-deploy/azure-app-service/deploy.sh` (Bash Version)

   - `/examples/tutorials/flow-deploy/azure-app-service/deploy.ps1` (PowerShell Version)

   Use the above deploy script to build and deploy the image.

Testing the Deployed Flow

Once your flow is deployed to Azure App Service, you can test it by sending a POST request to the endpoint or by browsing the test page. 

To test the flow:

1. Use a REST client like Postman or CURL to send a POST request to your endpoint

   - Sample endpoint: `https://honda-pf-99d9m.azurewebsites.net/score`

   - Make sure to set the Content-Type header to `application/json`

   - Include your request payload in the body of the POST request

2. **Test via the built-in test page** that comes with the deployment

   - Access the test page by navigating to your App Service URL in a browser

   - This provides a simple interface to test your flow without additional tools

The deployed flow exposes an API endpoint that follows the same interface patterns as when you test locally, making it straightforward to transition from development to production.

Conclusion

By following these steps, I was able to successfully deploy my Azure ML Prompt Flow to Azure App Service, creating a robust and scalable solution for my AI application. This approach provided the perfect balance of simplicity and functionality, allowing me to get my application into production quickly with minimal overhead.

The combination of Microsoft Copilot Studio for UI orchestration and Azure App Service for backend deployment created a powerful and flexible architecture that can be adapted to a variety of AI application scenarios.

Join the Conversation: Together We Learn

If you've found a better way to handle certain aspects of the deployment, please share your insights - I'm always looking to improve this workflow!

Unleash Your LLM Potential: Deploying an Azure ML Prompt Flow: A Practical Guide

Introduction

In the rapidly evolving landscape of Large Language Models (LLMs), efficiently deploying your AI applications is crucial. Today, I want to share my recent exploration into deploying Azure Machine Learning Prompt Flows, a powerful tool for streamlining the entire LLM application development lifecycle.

What is Azure ML Prompt Flow?

Azure ML Prompt Flow is more than just a development tool, it's a complete ecosystem designed to streamline the entire lifecycle of AI application development. 

Think of it as an orchestrator that lets you chain together prompts, Python scripts, data sources, and evaluation metrics in a structured pipeline.

My Recent Project: Diving into Deployment

Azure ML Prompt Flow promised to simplify the process, and I was eager to see if it lived up to the hype. It definitely did! It provided a structured way to build, test, and iterate on my LLM-powered application. My goal was simple: transform a promising AI prototype into a robust, production-ready application.

My Journey Through Azure Prompt Flow Deployment Strategies

Deploying an AI application is no longer a one-size-fits-all endeavor. My recent project with Azure Prompt Flow illuminated the complexity and flexibility of modern AI deployment strategies. Drawing directly from Microsoft's official documentation, I'll break down the four primary deployment approaches that can transform your AI project from a prototype to a production-ready solution.

Deployment Approaches: A Deep Dive

These are the four available approaches recommended by Microsoft:

**1. Deploy to Azure App Service**

This method offers a fully managed platform for hosting web applications. This approach is particularly compelling for developers seeking:

* Rapid deployment

* Minimal infrastructure management

* Easy scaling capabilities

* Simplified web application hosting

**2. Deploy a flow using Docker**

Docker provides containerization, enabling you to package your Prompt Flow and its dependencies into a portable container. Containerization through Docker offers unprecedented consistency and portability for your Prompt Flow applications:

The key capabilities or features

* Package entire application environment

* Ensure consistency across development and production

* Simplify dependency management

* Enable seamless migration between different infrastructure

**3. Deploy a flow using Kubernetes**

For applications demanding maximum scalability and reliability, Kubernetes emerges as the gold standard:

The Key Benefits:

* Advanced container orchestration

* Automatic scaling and load balancing

* High availability architecture

* Complex microservice management

**4. Deploy the Prompt Flow code to Azure AI Foundry**

Microsoft's Azure AI Foundry represents the next evolution in AI application deployment:

Azure AI Foundry is a newer offering that helps to provide a development environment that makes it easier to create and share AI solutions.

This option is great for collaborative development and sharing of AI solutions.

The key capabilities or features

* Integrated AI development environment

* Streamlined model management

* Enhanced collaboration tools

* Comprehensive AI solution lifecycle support

Choosing Your Deployment Strategy

Selecting the right approach depends on multiple factors:

**Project Complexity:**

* Simple web app → Azure App Service

* Consistent environment needs → Docker

* Enterprise-scale applications → Kubernetes

* Collaborative AI development → Azure AI Foundry

**Other Factors:**

* Scalability Requirements

* Team Expertise

* Infrastructure Constraints

* Performance Expectations

My two cents:

If you're looking to deploy Azure ML Prompt Flows, don't be afraid to experiment. Choose a deployment method that aligns with your needs and comfort level, and be prepared to get your hands dirty. The learning experience is worth it!

I'd love to hear about your experiences deploying Prompt Flows! Share your stories and tips in the comments below.

Happy deploying!!!

Webchat.JS / DirectLine API - how to clear the chat bot previous messages

 store.getState().activities = [];

For more details, please follow this post,
 
https://stackoverflow.com/questions/79487310/webchat-js-direct-line-api-copilot-studio-reset-the-chat-bot-messages

Elastic Search - Error -- curl: (52) Empty reply from server

in windows Curl command , the below  bulk import will throw the exception

C:\elasticsearch-8.12.2>curl -XPOST "https://localhost:9200/products/_bulk" -H "Content-Type: application/json" --data-binary "@products-bulk.json"   

Error:  curl: (52) Empty reply from server

Solution :

Use the cacert and -insecure and -u uid:pwd added to your curl command.
I tested and is working fine for me in Windows

curl --cacert config/certs/http_ca.crt --insecure -u elastic:l9RyDpzFJ7TBwhC06e9E -XPOST "https://localhost:9200/products/_bulk" -H "Content-Type: application/json" --data-binary "@products-bulk.json"

 

 

ATG BCC -- Remove struck Projects - SQL + RQL

 -- Removing locks of the project if any
delete from avm_asset_lock where workspace_id in
(select id from avm_devline where name in
(select workspace from epub_project where project_id = 'prj448130'));

-- Remove workspace
delete from avm_workspace where ws_id in
(select id from avm_devline where name in
(select workspace from epub_project where project_id = 'prj448130'));

-- Remove Devline
delete from avm_devline where name in
(select workspace from epub_project where project_id = 'prj448130');

-- delete history of the project
delete from EPUB_PR_HISTORY where project_id in
(select project_id from epub_project where project_id = 'prj448130');

-- delete history of the process
delete from EPUB_PROC_HISTORY where process_id in
(select process_id from epub_process where project = 'prj448130');

-- delete task information of process
delete from EPUB_PROC_TASKINFO where id in
(select process_id from epub_process where project = 'prj448130');

-- delete states of project (if any)
delete from EPUB_WORKFLOW_STRS where id in
(select ID from EPUB_IND_WORKFLOW where process_id in
(select process_id from epub_process where project = 'prj448130'));

delete  from EPUB_IND_WORKFLOW where process_id in
(select process_id from epub_process where project = 'prj448130');


-- delete the deployment
delete from epub_deployment where deployment_id in
(select deployment_id from epub_deploy_proj where project_id='prj448130');

-- delete the deployment data
delete from epub_deploy_proj where project_id='prj448130';

-- finally delete the process
delete from epub_process where project = 'prj448130';

Invalidate Caches

1) /dyn/admin/nucleus/atg/deployment/DeploymentRepository/
2) /dyn/admin/nucleus//atg/epub/PublishingRepository/
3) /atg/epub/version/VersionManagerRepository/

Now navigate to :
/atg/epub/PublishingRepository/
if the project name exists in BCC Home, check this RQL , if it returns data

<print-item item-descriptor="project" id="prj252141" />
<print-item item-descriptor="process" id="prc252141" />

if so, remove them in the below reverse order

<remove-item item-descriptor="process" id="prc444019" />
<remove-item item-descriptor="project" id="prj444019" />


<print-item item-descriptor="project" id="prj252141" />
<print-item item-descriptor="process" id="prc252141" />

JSP - debugging - VirtualContextRootInterceptor

Some times, you may not see any error in the logs, even though there are JSP errors.
or it may be difficult to find where exactly the error is being thrown from the JSP/DSP pages

The below components becomes handy in this stage, enable the logging debug on the component, then watch your logs.

/atg/dynamo/servlet/dafpipeline/VirtualContextRootInterceptor/

Alternatively, you can watch your start up logs also.

Internet explorer + Developer tools + Sig web intergration.

We are trying to intergrate the New Topaz sig web with our ATG application running on Weblogic.

Issues : The UI SIGNATURE capture will works in IE only if Developer tools (  F12 ) is enabled. if not enabled it wont work, especially the below
JS, which has all the required method calls

http://www.sigplusweb.com/SigWebTablet.js

Reason : The above JS has AJAX js  GET request. and IE caches it and then the next request never reaches the
Sigweb engine.

Resolution:

1) Using a unique query string parameter (such as a unix timestamp) on each request,
so as to make each request URL unique, thereby preventing caching.



Why F12 works : Looks like I have  "always refresh from server" option enabled in F12.
Other browsers don't cache Ajax calls, but IE does

IVY and JRebel -

I am seeing two very useful tools which can ease your local development setup and build process in your ATG Project.

1. JRebel - automated build, similar to file-sync , plus it will sync up your .java file changes. No need to build your ear after every Java change.
    One build daily, and use JRebel, which will take care of your Java changes. (ie Reload Code Changes Instantly)
    Limitations : ATG properties files changes seems not picking up.

2. Ivy - Ivy is a tool for managing / resolving  project dependencies.(You dont need to worry about adding project to classpath/ creating  userdefined libraries and all ).

Ivy is open source and released under a very permissive Apache License.
JRebel is Licensed.

Setting up Performance Monitor And Testing with HTTPERF

Setting up Performance Monitor: ATG- Endeca Intergration Experience Manager Performance.

A quick write up.

Install Cygwin and install httperf.
https://code.google.com/p/httperf/  -- 

Configure HttPerf in Cygwin. How to ?? Refer below link
http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/2006-August/000291.html  -- this might take some time. Better install all in one shot.[gcc, make …]

Note: If you need to test from your local windows ONLY. [Your can use any Linux/Mac envr if you have httperf installed ]

akg@akg-hp-envy-15-notebook-pc:~$ sudo apt-get install httperf -- from your Unix shell

Enabling Performance Monitoring in your ATG Components

Pre-Check
1. Navigate to /atg/dynamo/admin/en/performance-monitor-config.jhtml
   Make the button Enabled to TIME or MEMORY

2. Click [Reset Data]

3. Hit the URLs over the browser (use httperf for load testing)

   eg : http://myServer.vci.CustomCatriges.com:8710/mybusiness/b/search/?Ntt=iphone&format=json
        http://myServer.vci.CustomCatriges.com:8710/mybusiness/b/search/?Ntt=samsung


4. Navigate to http://<localhost:7001>/dyn/admin/atg/dynamo/admin/en/performance-monitor.jhtml



Enabling Performance Monitor: in your custom catridges, my example below

\CORE\src\com\endeca\infront\cartridge\CustomCatrigesRecordTypeResultsListHandler.java
\CORE\src\com\endeca\infront\cartridge\CustomCatrigesRecordTypeMenuHandler.java
\CORE\src\com\endeca\infront\cartridge\CustomCatrigesRecordTypeResultsListHandler.java

Refer the above classes and cross check those indicators in ATG url in #4.You should be able to see the time in msec.

PerformanceMonitor.startOperation("CustomCatrigesRecordTypeResultsListHandler [preprocess]", "preprocess"); ## Add these indicators - from my code snippet.

PerformanceMonitor.endOperation("CustomCatrigesRecordTypeResultsListHandler [preprocess]", "preprocess");

Also , the ootb components can be cleaned, and compare the Assembler Tools Average execution Time.

    Operation    Number of Executions    Average Execution Time (msec)    Minimum Execution Time (msec)    Maximum Execution Time (msec)    Total Execution Time (msec)
    AssemblerTools: invoke assembler     200                2262     121      7797    452472                /mybusiness/search?q=LG
    CustomCatrigesRecordTypeResultsListHandler [preprocess]     216    0    0    0    0                /mybusiness/search?q=Refurbished



RecordSpotlightConfig.properties
ResultsListConfig.properties"       

JSP content redering as it is - Weblogic + ATG 10.2

I have an issue with my ATG store application.
Content of the jsp is written to the browser instead of rendering it.

http://localhost:7001/ourbusiness/b?N=129&Ept=acc
<!-- Result in Browser  -->

<dsp:page>
    <dsp:include page="/global/endeca-helper-component.jsp" />
</dsp:page>

<!-- Result in Browser ENDS  -->


AdminServer.Log is below

odel.Record@7ad68151, com.endeca.infront.cartridge.model.Record@18c42bd8, com.endeca.infront.cartridge.model.Record@771a7e9b, com.endeca.infront.cartridge.model.Record@6c6e2f33, com.endeca.
tridge.model.Record@35dcf4e2, com.endeca.infront.cartridge.model.Record@3d513d0e], precomputedSorts=[]}]}]}]}
**** debug      Tue Oct 21 16:28:27 CDT 2014    1413926907491   /atg/endeca/assembler/AssemblerPipelineServlet  No site base URL found, initializing dispatcher with path: /cartridges/PageSl
.jsp
<Oct 21, 2014 4:28:43 PM CDT> <Info> <Health> <BEA-310002> <176176670f the total memory in the server is free>
Can you please throw some lights to this issue?



Solution :

The below block has been missing in eBiz-CE1‘s Config.xml -Weblogic server and highlighted is a key for an issue as missing . 
<web-app-container>
    <relogin-enabled>false</relogin-enabled>
    <allow-all-roles>false</allow-all-roles>
    <filter-dispatched-requests-enabled>false</filter-dispatched-requests-enabled>
    <overload-protection-enabled>false</overload-protection-enabled>
    <x-powered-by-header-level>SHORT</x-powered-by-header-level>
    <mime-mapping-file>./config/mimemappings.properties</mime-mapping-file>
    <optimistic-serialization>false</optimistic-serialization>
    <rtexprvalue-jsp-param-name>false</rtexprvalue-jsp-param-name>
    <client-cert-proxy-enabled>false</client-cert-proxy-enabled>
    <http-trace-support-enabled>false</http-trace-support-enabled>
    <weblogic-plugin-enabled>true</weblogic-plugin-enabled>
    <auth-cookie-enabled>true</auth-cookie-enabled>
    <wap-enabled>false</wap-enabled>
    <post-timeout-secs>30</post-timeout-secs>
    <max-post-time-secs>-1</max-post-time-secs>
    <max-post-size>-1</max-post-size>
    <work-context-propagation-enabled>true</work-context-propagation-enabled>
    <jsp-compiler-backwards-compatible>false</jsp-compiler-backwards-compatible>
    <show-archived-real-path-enabled>false</show-archived-real-path-enabled>
    <change-session-id-on-authentication>true</change-session-id-on-authentication>
  </web-app-container>



Thanks to Robert Sebastian for the fix:

10.2 BCC showing AmlvmCatFacet.display tab in category

10.2 BCC showing AmlvmCatFacet.display tab in category


Issue: When I am creating a new category  in bcc version atg10.2 , then New there is a tab AmlvmCatFacet.display

Fix : This seems to be an issue with ATG upgrade:

Steps:

Navigate to  :    
        http://<localhost:7001>/dyn/admin/nucleus//atg/web/viewmapping/ViewMappingRepository/
        <remove-item item-descriptor="itemViewMapping" id="AmIvmCatFacet"/>
              Invalidate ViewMappingRepository  Cache

ATG- Endeca, how to exclude Child-SKUs from Indexing

I found this solution working..
http://www.spltech.co.uk/blog/atg/excluding-unwanted-skus-from-endeca

Endeca Manipulators

This feature of Endeca allows you to manipulate/alter/tweak the data which is indexed from
ATG.

Steps:
1. Create the Java manipulators
    -- http://docs.oracle.com/cd/E29587_01/PlatformServices.60x/ps_cadk  /src/rcdk_app_code_java_manip.html

2. Compile it and export to jar(can happen via build)

3.  Edit your exp manager developer studio,
4. Add this in the endeca pipeline process.
5. Trigger Base line indexing.
6. Watch for the "Forge" step, if error it will throw u the error.




########### Sample error Logs
Caused by com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

com.endeca.soleng.eac.toolkit.component.BatchComponent run - Batch component  'Forge' failed. Refer to component
logs in /home/homeuser/ENDECA/endeca/apps/atgsid/./logs/forges/Forge on host ITLHost.



[08.07.14 11:53:55] INFO: Released lock 'update_lock'.

[homeuser@zltv5746 control]$ more /home/homeuser/ENDECA/endeca/apps/atgsid/./logs/forges/Forge/

Edf.Pipeline.RecordPipeline.JavaManipulator.R3_Price_Split_JavaManipulator.log  Forge.start.log

Forge.log

[homeuser@zltv5746 control]$ more /home/homeuser/ENDECA/endeca/apps/atgsid/./logs/forges/Forge/Edf.Pipeline.RecordPipeline.JavaManipulator.R3_Price_Split_JavaManipulator.log

RelativePathFileHandler did not find a system property named endeca.project.dir or an environment variable named ENDECA_PROJECT_DIR. Defaulting to current working directory.

[08.07.14 11:53:53] SEVERE: sku.promoPrice passthrough not specified; aborting



Caused by com.endeca.edf.adapter.AdapterException

com.atg.sid.endeca.pipeline.manipulators.ATGEndecaPriceManipulator execute - sku.promoPrice passthrough not specified; aborting


##########

Endeca Baseline Update warining -- WARNING: Failed to obtain lock

While baseline update, some time you can end-up with the below issue

C:\Endeca\Apps\atmsid\control>baseline_update.bat
[07.29.14 16:45:44] INFO: Checking definition from AppConfig.xml against existing EAC provisioning.
[07.29.14 16:45:46] INFO: Definition has not changed.
[07.29.14 16:45:46] INFO: Starting baseline update script.
[07.29.14 16:45:46] WARNING: Failed to obtain lock.

Solution :

eaccmd remove-all-flags --app atmsid      /*** -- where atmsid     is ur application name ****/

Endeca Controller - start and stop Endeca Server - Write in progress with generation

Concurrent Update error : 

ispatcherThread.java:178)
**** Error      Thu Jun 19 08:13:03 PDT 2014    1403190783340   /atg/commerce/endeca/index/CategoryToDimensionOutputConfig
**** info       Thu Jun 19 08:13:03 PDT 2014    1403190783355   /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin    Indexing process cancelled, Endeca says: atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 42
**** Error      Thu Jun 19 08:13:03 PDT 2014    1403190783357   /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin    ---     atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 42

 The below script can be use to start and stop the three Endeca Services.
Also clear the logs before restarting.

##############
#!/bin/bash


########################################################
# #     Endeca Clean and Start Scripts               ###
# #                                                  ###
# #     Author : Anilal KG                           ###
# #                                                  ###
# #     Version  : 0.1                               ###
########################################################


SCRIPT=endeca
ENDECA_INSTALL_BASE=/path/to/endeca

PS_BIN=${ENDECA_INSTALL_BASE}/PlatformServices/6.1.4/tools/server/bin
TF_BIN=${ENDECA_INSTALL_BASE}/ToolsAndFrameworks/3.1.2/server/bin
CAS_BIN=${ENDECA_INSTALL_BASE}/CAS/3.1.2.1/bin
APP_LOG=${ENDECA_INSTALL_BASE}/apps/$1/logs/  ## path to endeca apps logs

if [ -z "$1" ]
    then
         echo ""
            echo "Usage: "
            echo ""
            echo "  sh endecacontroller.sh  <appname> "
            exit
fi

if [ ! -d "$APP_LOG" ]; then
    printf "\n The ENDECA application -- $1 -- does not exists .Please check once again \n"
    exit
fi

echo ""
echo "Stopping CAS  ".${CAS_BIN}
${CAS_BIN}/cas-service-shutdown.sh


echo ""
echo "Stopping Tools And Frameworks  ".${TF_BIN}
${TF_BIN}/shutdown.sh


##
echo ""
echo "Stopping PlatformServices  ".${PS_BIN}
${PS_BIN}/shutdown.sh 


## clearing logs
echo ""
echo "Clearing the application logs .."
echo $APP_LOG
rm -rf $APP_LOG/*.*



echo ""
echo "Starting Platform Services  "
${PS_BIN}/startup.sh 


echo ""
echo "Starting Tools And Frameworks  "
${TF_BIN}/startup.sh


echo ""
echo "Starting CAS  "
${CAS_BIN}/cas-service.sh &


echo ""
echo ""
echo "Status of the ports listening -double check manually --Listenting for ports..."
echo ""
echo ""

netstat -plnt | grep :::8


##############

SQL Query to find the table name which contains specific data

Requirement :  when you don't know what table or what column in your schema have the data.

This example, generates the table name / column name which contains the Data - 'samsung';

Thanks to Mohan for sending this Query. Worked fine for me hence posting .

######################

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\ag4063>sqlplus /nolog

SQL*Plus: Release 11.2.0.1.0 Production on Thu Jun 12 18:22:49 2014

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

SQL> connect sid_contenta
Enter password:
Connected.
SQL>

#####################################

spool outFile.txt

set serveroutput ON size 1000000

declare
srch_string varchar2(100) := 'samsung';
str varchar2(4000);
PROCEDURE get_column_list(tabname in varchar2) IS
begin
for c1 in(select column_name from user_tab_columns where table_name=tabname and data_type like '%CHAR%')
LOOP
str := str || 'upper('||c1.column_name||') like upper(''%'||srch_string||'%'') or ';
END LOOP;
str := rtrim(str, 'or ');
END get_column_list;
begin
for c2 in(select table_name from user_tables) LOOP
str := 'select * from '||c2.table_name||' where ';
get_column_list(c2.table_name);
dbms_output.put_line(str||';');
END LOOP;
end;
/

################################################
spool off
###############################################