Setting up Performance Monitor And Testing with HTTPERF

Setting up Performance Monitor: ATG- Endeca Intergration Experience Manager Performance.

A quick write up.

Install Cygwin and install httperf.
https://code.google.com/p/httperf/  -- 

Configure HttPerf in Cygwin. How to ?? Refer below link
http://www.hpl.hp.com/hosted/linux/mail-archives/httperf/2006-August/000291.html  -- this might take some time. Better install all in one shot.[gcc, make …]

Note: If you need to test from your local windows ONLY. [Your can use any Linux/Mac envr if you have httperf installed ]

akg@akg-hp-envy-15-notebook-pc:~$ sudo apt-get install httperf -- from your Unix shell

Enabling Performance Monitoring in your ATG Components

Pre-Check
1. Navigate to /atg/dynamo/admin/en/performance-monitor-config.jhtml
   Make the button Enabled to TIME or MEMORY

2. Click [Reset Data]

3. Hit the URLs over the browser (use httperf for load testing)

   eg : http://myServer.vci.CustomCatriges.com:8710/mybusiness/b/search/?Ntt=iphone&format=json
        http://myServer.vci.CustomCatriges.com:8710/mybusiness/b/search/?Ntt=samsung


4. Navigate to http://<localhost:7001>/dyn/admin/atg/dynamo/admin/en/performance-monitor.jhtml



Enabling Performance Monitor: in your custom catridges, my example below

\CORE\src\com\endeca\infront\cartridge\CustomCatrigesRecordTypeResultsListHandler.java
\CORE\src\com\endeca\infront\cartridge\CustomCatrigesRecordTypeMenuHandler.java
\CORE\src\com\endeca\infront\cartridge\CustomCatrigesRecordTypeResultsListHandler.java

Refer the above classes and cross check those indicators in ATG url in #4.You should be able to see the time in msec.

PerformanceMonitor.startOperation("CustomCatrigesRecordTypeResultsListHandler [preprocess]", "preprocess"); ## Add these indicators - from my code snippet.

PerformanceMonitor.endOperation("CustomCatrigesRecordTypeResultsListHandler [preprocess]", "preprocess");

Also , the ootb components can be cleaned, and compare the Assembler Tools Average execution Time.

    Operation    Number of Executions    Average Execution Time (msec)    Minimum Execution Time (msec)    Maximum Execution Time (msec)    Total Execution Time (msec)
    AssemblerTools: invoke assembler     200                2262     121      7797    452472                /mybusiness/search?q=LG
    CustomCatrigesRecordTypeResultsListHandler [preprocess]     216    0    0    0    0                /mybusiness/search?q=Refurbished



RecordSpotlightConfig.properties
ResultsListConfig.properties"       

JSP content redering as it is - Weblogic + ATG 10.2

I have an issue with my ATG store application.
Content of the jsp is written to the browser instead of rendering it.

http://localhost:7001/ourbusiness/b?N=129&Ept=acc
<!-- Result in Browser  -->

<dsp:page>
    <dsp:include page="/global/endeca-helper-component.jsp" />
</dsp:page>

<!-- Result in Browser ENDS  -->


AdminServer.Log is below

odel.Record@7ad68151, com.endeca.infront.cartridge.model.Record@18c42bd8, com.endeca.infront.cartridge.model.Record@771a7e9b, com.endeca.infront.cartridge.model.Record@6c6e2f33, com.endeca.
tridge.model.Record@35dcf4e2, com.endeca.infront.cartridge.model.Record@3d513d0e], precomputedSorts=[]}]}]}]}
**** debug      Tue Oct 21 16:28:27 CDT 2014    1413926907491   /atg/endeca/assembler/AssemblerPipelineServlet  No site base URL found, initializing dispatcher with path: /cartridges/PageSl
.jsp
<Oct 21, 2014 4:28:43 PM CDT> <Info> <Health> <BEA-310002> <176176670f the total memory in the server is free>
Can you please throw some lights to this issue?



Solution :

The below block has been missing in eBiz-CE1‘s Config.xml -Weblogic server and highlighted is a key for an issue as missing . 
<web-app-container>
    <relogin-enabled>false</relogin-enabled>
    <allow-all-roles>false</allow-all-roles>
    <filter-dispatched-requests-enabled>false</filter-dispatched-requests-enabled>
    <overload-protection-enabled>false</overload-protection-enabled>
    <x-powered-by-header-level>SHORT</x-powered-by-header-level>
    <mime-mapping-file>./config/mimemappings.properties</mime-mapping-file>
    <optimistic-serialization>false</optimistic-serialization>
    <rtexprvalue-jsp-param-name>false</rtexprvalue-jsp-param-name>
    <client-cert-proxy-enabled>false</client-cert-proxy-enabled>
    <http-trace-support-enabled>false</http-trace-support-enabled>
    <weblogic-plugin-enabled>true</weblogic-plugin-enabled>
    <auth-cookie-enabled>true</auth-cookie-enabled>
    <wap-enabled>false</wap-enabled>
    <post-timeout-secs>30</post-timeout-secs>
    <max-post-time-secs>-1</max-post-time-secs>
    <max-post-size>-1</max-post-size>
    <work-context-propagation-enabled>true</work-context-propagation-enabled>
    <jsp-compiler-backwards-compatible>false</jsp-compiler-backwards-compatible>
    <show-archived-real-path-enabled>false</show-archived-real-path-enabled>
    <change-session-id-on-authentication>true</change-session-id-on-authentication>
  </web-app-container>



Thanks to Robert Sebastian for the fix:

10.2 BCC showing AmlvmCatFacet.display tab in category

10.2 BCC showing AmlvmCatFacet.display tab in category


Issue: When I am creating a new category  in bcc version atg10.2 , then New there is a tab AmlvmCatFacet.display

Fix : This seems to be an issue with ATG upgrade:

Steps:

Navigate to  :    
        http://<localhost:7001>/dyn/admin/nucleus//atg/web/viewmapping/ViewMappingRepository/
        <remove-item item-descriptor="itemViewMapping" id="AmIvmCatFacet"/>
              Invalidate ViewMappingRepository  Cache

ATG- Endeca, how to exclude Child-SKUs from Indexing

I found this solution working..
http://www.spltech.co.uk/blog/atg/excluding-unwanted-skus-from-endeca

Endeca Manipulators

This feature of Endeca allows you to manipulate/alter/tweak the data which is indexed from
ATG.

Steps:
1. Create the Java manipulators
    -- http://docs.oracle.com/cd/E29587_01/PlatformServices.60x/ps_cadk  /src/rcdk_app_code_java_manip.html

2. Compile it and export to jar(can happen via build)

3.  Edit your exp manager developer studio,
4. Add this in the endeca pipeline process.
5. Trigger Base line indexing.
6. Watch for the "Forge" step, if error it will throw u the error.




########### Sample error Logs
Caused by com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

com.endeca.soleng.eac.toolkit.component.BatchComponent run - Batch component  'Forge' failed. Refer to component
logs in /home/homeuser/ENDECA/endeca/apps/atgsid/./logs/forges/Forge on host ITLHost.



[08.07.14 11:53:55] INFO: Released lock 'update_lock'.

[homeuser@zltv5746 control]$ more /home/homeuser/ENDECA/endeca/apps/atgsid/./logs/forges/Forge/

Edf.Pipeline.RecordPipeline.JavaManipulator.R3_Price_Split_JavaManipulator.log  Forge.start.log

Forge.log

[homeuser@zltv5746 control]$ more /home/homeuser/ENDECA/endeca/apps/atgsid/./logs/forges/Forge/Edf.Pipeline.RecordPipeline.JavaManipulator.R3_Price_Split_JavaManipulator.log

RelativePathFileHandler did not find a system property named endeca.project.dir or an environment variable named ENDECA_PROJECT_DIR. Defaulting to current working directory.

[08.07.14 11:53:53] SEVERE: sku.promoPrice passthrough not specified; aborting



Caused by com.endeca.edf.adapter.AdapterException

com.atg.sid.endeca.pipeline.manipulators.ATGEndecaPriceManipulator execute - sku.promoPrice passthrough not specified; aborting


##########

Endeca Baseline Update warining -- WARNING: Failed to obtain lock

While baseline update, some time you can end-up with the below issue

C:\Endeca\Apps\atmsid\control>baseline_update.bat
[07.29.14 16:45:44] INFO: Checking definition from AppConfig.xml against existing EAC provisioning.
[07.29.14 16:45:46] INFO: Definition has not changed.
[07.29.14 16:45:46] INFO: Starting baseline update script.
[07.29.14 16:45:46] WARNING: Failed to obtain lock.

Solution :

eaccmd remove-all-flags --app atmsid      /*** -- where atmsid     is ur application name ****/

Endeca Controller - start and stop Endeca Server - Write in progress with generation

Concurrent Update error : 

ispatcherThread.java:178)
**** Error      Thu Jun 19 08:13:03 PDT 2014    1403190783340   /atg/commerce/endeca/index/CategoryToDimensionOutputConfig
**** info       Thu Jun 19 08:13:03 PDT 2014    1403190783355   /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin    Indexing process cancelled, Endeca says: atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 42
**** Error      Thu Jun 19 08:13:03 PDT 2014    1403190783357   /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin    ---     atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 42

 The below script can be use to start and stop the three Endeca Services.
Also clear the logs before restarting.

##############
#!/bin/bash


########################################################
# #     Endeca Clean and Start Scripts               ###
# #                                                  ###
# #     Author : Anilal KG                           ###
# #                                                  ###
# #     Version  : 0.1                               ###
########################################################


SCRIPT=endeca
ENDECA_INSTALL_BASE=/path/to/endeca

PS_BIN=${ENDECA_INSTALL_BASE}/PlatformServices/6.1.4/tools/server/bin
TF_BIN=${ENDECA_INSTALL_BASE}/ToolsAndFrameworks/3.1.2/server/bin
CAS_BIN=${ENDECA_INSTALL_BASE}/CAS/3.1.2.1/bin
APP_LOG=${ENDECA_INSTALL_BASE}/apps/$1/logs/  ## path to endeca apps logs

if [ -z "$1" ]
    then
         echo ""
            echo "Usage: "
            echo ""
            echo "  sh endecacontroller.sh  <appname> "
            exit
fi

if [ ! -d "$APP_LOG" ]; then
    printf "\n The ENDECA application -- $1 -- does not exists .Please check once again \n"
    exit
fi

echo ""
echo "Stopping CAS  ".${CAS_BIN}
${CAS_BIN}/cas-service-shutdown.sh


echo ""
echo "Stopping Tools And Frameworks  ".${TF_BIN}
${TF_BIN}/shutdown.sh


##
echo ""
echo "Stopping PlatformServices  ".${PS_BIN}
${PS_BIN}/shutdown.sh 


## clearing logs
echo ""
echo "Clearing the application logs .."
echo $APP_LOG
rm -rf $APP_LOG/*.*



echo ""
echo "Starting Platform Services  "
${PS_BIN}/startup.sh 


echo ""
echo "Starting Tools And Frameworks  "
${TF_BIN}/startup.sh


echo ""
echo "Starting CAS  "
${CAS_BIN}/cas-service.sh &


echo ""
echo ""
echo "Status of the ports listening -double check manually --Listenting for ports..."
echo ""
echo ""

netstat -plnt | grep :::8


##############

SQL Query to find the table name which contains specific data

Requirement :  when you don't know what table or what column in your schema have the data.

This example, generates the table name / column name which contains the Data - 'samsung';

Thanks to Mohan for sending this Query. Worked fine for me hence posting .

######################

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\ag4063>sqlplus /nolog

SQL*Plus: Release 11.2.0.1.0 Production on Thu Jun 12 18:22:49 2014

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

SQL> connect sid_contenta
Enter password:
Connected.
SQL>

#####################################

spool outFile.txt

set serveroutput ON size 1000000

declare
srch_string varchar2(100) := 'samsung';
str varchar2(4000);
PROCEDURE get_column_list(tabname in varchar2) IS
begin
for c1 in(select column_name from user_tab_columns where table_name=tabname and data_type like '%CHAR%')
LOOP
str := str || 'upper('||c1.column_name||') like upper(''%'||srch_string||'%'') or ';
END LOOP;
str := rtrim(str, 'or ');
END get_column_list;
begin
for c2 in(select table_name from user_tables) LOOP
str := 'select * from '||c2.table_name||' where ';
get_column_list(c2.table_name);
dbms_output.put_line(str||';');
END LOOP;
end;
/

################################################
spool off
###############################################

Version issues with DSP Taglib

Error Log:

1398356350603               /atg/registry/webappregistry/ServletContextWebAppRegistry       
The version of the DSP Taglib from the web-app "BCC" found
within "zip:C:/ATG/ATG10.2/EAR/BCC.ear/atglib/_DAS_slib_sclasses.jar!/"
did not match the expected version. Please update all copies of the
 DSP Taglib with the version in $DYNAMO_HOME/DAS/taglib/dspjspTaglib/1.0/lib/
and re-assemble your application. Enable loggingDebug on this component
and restart to see which resource differs.


Fix: To fix above issue, replace taglibs of BCC/j2ee-apps/BCCWeb/BCCWeb.war/WEB-INF/lib from C:\ATG\ATG10.2\DAS\taglib\dspjspTaglib\1.0

Note: This fix I am yet to test it in my local. Please try and update if it works.

Intergrating ATG-Endeca for Indexing

Note: This is with Version ATG 10.2 onwards Only


Step : 1

You have to have your below component configured to use your PCOC file like below

/atg/commerce/search/ProductCatalogOutputConfig/

definitionFile=/atg/commerce/endeca/index/product-sku-output-config.xml

Sample file Attached:

Step : 2

Your combined definition file should be a union of below xmls(OOTB+Custom)
Source files   

\config\config.jar/atg/commerce/endeca/index/product-sku-output-config.xml
\DCS\Endeca\Index\config\config.jar/atg/commerce/endeca/index/product-sku-output-config.xml
\DCS\Endeca\Index\SKUIndexing\config\config.jar/atg/commerce/endeca/index/product-sku-output-config.xml


Step : 3

Build level changes : Add the below modules in your ANT build.properties file
for eg:
my.ear.modules=DafEar.Admin,DCS.Endeca.Index.Versioned,DCS.Endeca.Index,DAF.Endeca.Index,DAF.Endeca.Assembler,DCS.Endeca.Index.SKUIndexing

Verification :  Once your build is up, you should be able to see this componenet via dyn/admin
http://mylocalserver:7001/dyn/admin/nucleus/atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/

Step : 4

Make these config layer changes.Override bases on your environments.

/atg/endeca/ApplicationConfiguration
baseApplicationName: myendecaapp
keyToApplicationName: default=myendecaapp,en=myendecaapp
workbenchHostName: myendeca.envr.server.name.com
workbenchPort: 8006

/atg/endeca/index/IndexingApplicationConfiguration
CAS Hostname: myendeca.envr.server.name.com
CAS Port: 8500

EAC Hostname: myendeca.envr.server.name.com
EAC Port: 8888

/atg/endeca/assembler/AssemblerApplicationConfiguration
currentMdexHostName: myendeca.envr.server.name.com
currentMdexPort: 16000

defaultMdexHostName: myendeca.envr.server.name.com
defaultMdexPort: 16000
applicationKeyToMdexHostAndPort: en= myendeca.envr.server.name.com:16000


Step 5: (Optional)

If you are setting up your endeca indexing from your BCC instance, you need to override the below component

Set the OnPubServer flag to false for the following components

/atg/commerce/search/ProductCatalogOutputConfig
/atg/commerce/endeca/index/CategoryToDimensionOutputConfig


Step 6:

Triggering Indexing from Dyn Admin:

Navigate : http://mylocalserver:7001/dyn/admin/nucleus/atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/

Trigger the BaseLine Index Button.Watch your ATG logs for issues.

#################################
Reference  : http://docs.oracle.com/cd/E23095_01/Platform.93/ATGSearchIntegration/html/s0202specifyingthedefinitionfileandth01.html

How to set up your local BCC.

How to set up your local BCC.

I thought to update here what I did for setting up my ATG-BCC up and running in my local box.


Steps I did:

1. Imported the db which is running, for a working instance.
2. Cleaned up the PUB schema -- refer the below links
    http://atgtech.blog.com/2013/10/15/159/

3.  delete from EPUB_COLL_WORKFLOW
     delete from EPUB_WORKFLOW_INFO
     delete from EPUB_WF_SERVER_ID
     commit;

4. Restart the server,
  4.1.1  select * from EPUB_WF_SERVER_ID   // in your pub schema
             this should show you the server id of your local machine. Important !!!!

5. Login to BCC with the credentials of the database as in step #1.

------------------------

Thinks to ensure:

There should be role assigned properly for the user "Admin"

To verify it:

Navigate to : http:/localhost:7001/dyn/admin/nucleus/atg/userprofiling/InternalProfileRepository/

<print-item item-descriptor="user" id="portal-admin" />

------ Printing item with id: portal-admin
<add-item item-descriptor="user" id="portal-admin">
  <set-property name="securityStatus"><![CDATA[EXPLICIT-SIGNIN]]></set-property>
  <set-property name="email"><![CDATA[anilal.testing@tester.com]]></set-property>
  <set-property name="password"><![CDATA[tes]]></set-property>
  <!-- rdonly   <set-property name="accessRights"><![CDATA[previewuserfull,seotagsfull,mediaread,sitecatalogread,sitecatalogfull,promotionsfull,tgtsgmtfull,storesfull,siteadminfull,pricelistsfull,siteadminread,externaluserfull,mediafull,storecontentread,internaluserfull]]></set-property>  -->

  <set-property name="roles"><![CDATA[internalUserAdministrationRole,epubSuperAdmin,merchandiserRole,promotionsAdministratorRole,portal-admin,personalizationRole,siteAdminRole]]></set-property>
  <set-property name="login"><![CDATA[admin]]></set-property>

</add-item>


Make sure that you have these roles set up.
epubSuperAdmin -- should exists for the user admin for the property "Roles"