Multiple 407 Status Codes when reviewing a Client-Side Capture

 

When looking at a Client-Side capture (either with Fidddler2 or with RealiTea Viewer), at some sites we see the client is making multiple requests for a page, or resource. The sequence is

1. Request for /resource.do
2. Response Status Code 407
3. Request for /resource.do
4. Response Status Code 407
5. Request for /resource.do
6. Response code 200 (and the page is returned)

 

This pattern indicates that there is a proxy between the client and the web server, and the proxy is configured to use a Challenge/Response Authentication scheme. Here are some additional details for that same sequence above:

1. Request for /resource.do
• The browser sends no authentication headers in this request
2. Response Status Code 407
• The proxy responds with a “Proxy Authentication Required” Status code and also tell the web server what kind of authentication is requried, eitehr basic or Challenge/Response
3. Request for /resource.do
• The browser sends the request again, this time with a HTTP header that tells the proxy it wants to use Challenge/Response
4. Response Status Code 407
• The proxy responds with the 407 status code, and additional it includes the Challenge information
5. Request for /resource.do
• The browser sends the request again, this time with a HTTP header that includes the Response to the Challenge
• The proxy lets this request through to the web server
6. Response code 200 (and the page is returned
• The server returns the response to the proxy, which passes it back to the browser.

 

This article by MSDN has further details and  nice explanation http://msdn.microsoft.com/en-us/library/windows/desktop/aa383144(v=vs.85).aspx

Windows Server Setup for Tealeaf Servers

If you are setting up brand new Windows Server 2008 R2 systems for Tealeaf servers, here are the settings and programs you will want to configure. These assume the servers will be in a Workgroup, with no ACtive Directory. Remember – you cannot use a Domain Controller as a tealeaf server, the tealeaf software will not run on a Domain Controller.

Common to All Servers

• Configure the SAS drive that comes with the server so the available drive space is dedicated to C: Drive.
• Install Windows Server 2008 R2 Enterprise OS to the C: Drive.
• Provide information for Workgroup, Server Name, Local Administrator Name when asked
• For the local administrator, use TBD/TBD
• Place all servers in a workgroup TBD (one suggestion would be “Tealeafarm”)
• Provide information for IP Address, Default Gateway, when asked
• Assign the server name and IP address per the corporate standard and communication spreadsheet
• Default Gateway is TBD
• After installation completes, run Windows Updates, then reboot
• Disable the UAC for local administrators
• Disable User Access Control
• Start –> Settings –> Control Panel –> User Accounts –> Turn User Account Control  off
• Run this command (or edit registry by hand) to add a special registry key to make administrators true administrator: C:\Windows\System32\cmd.exe /k %windir%\System32\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /
• Set the following Windows Features (From Control panel add/remove programs->Windows Features)
• Enable Remote Desktop
• Start –> Settings –> Control Panel –> Programs—>Turn Windows Features On –> TBD
• Download, unblock, and install the .Net 3.5 Libraries
• TBD
• Configure a NTP Source
• Windows Time Service and Resulting Internet Communication in Windows Server 2008
• A list of the Simple Network Time Protocol (SNTP) time servers that are available on the Internet
• How to enable and use a local Network Time Protocol (NTP) server
• Configure Windows PowerShell as follows
• Set the Execution policy to unrestricted and enable PS-Remoting by opening a PowerShell Command Prompt and entering the following commands:

Set-ExecutionPolicy  Unrestricted
Enable-PSRemoting -Force

 

• Install the following additional SW for productivity. Install all to their default locations




• GreenShot (Screen Capture) Greenshot V 8.0.0627 Download


• Ditto (Clipboard manager) Ditto V3.17.0.17 Download


• Notepad++ (Enhanced Editor) Notepad++ v5.9.5 Download

HBR Servers (These will communicate with the PCA servers)

• Configure the SAN drive (one LUN, one drive, one TB) as drive D:


• Install the following additional SW for productivity. Install all to their default locations


• Windows Secure File Copy (WinSCP)  WinSCP  V4.3.5 Download


• Putty (SSH for Windows) Putty (Latest Version) Download

Canister Servers

• Configure the SAN drives (two LUNS, two drives, 1 TB each) as drives D: and E:

Image Server

• Configure the SAN drive (one LUN, one drive, one TB) as drive D:

Portal/Reporting Servers

• Configure the SAN drives (two LUNS, two drives, 1 TB each) as drives D: and E:


• Enable and Configure IIS




• 




• Install and Configure SQL Server 2008 R2 Enterprise Version




• TBD

How to update the Robot Detection files and lists

I’ve had a some tealeaf administrators ask me where to find the instructions for updating the Robot Detection and Mobile Devices list, so I decided to post my answers here for anyone else interested.

Start by getting the full documentation set for Tealeaf, if you don’t already have it. It’s a great way to have all the tealeaf doc in a single place.

You can get the full documentation set from community.tealeaf.com. If you don’t have a login setup there yet, I’d suggest you get a login there and explore it – there is a ton of resources, and a pretty active community forum where you can post questions and get answers. Once you have logged in, click the Online Help link. The down the left nav bar, you will find Product Docs. Click the highest release number you see visible, and you will see a screen similar to the following, which is the top-level menu for all the documents for a specific release. I’ve highlighted the link that gets you a zip file of all the documents in PDF form.

 

Within this zip, open the file “cxImpact_Administration_Manual_7.2.pdf”

Find Chapter 10. APPENDIX: MANAGING USER AGENTS, and read this chapter for the theory on how bot and mobile detection works.

Next, from that same zip file, open the document “CX_Installation_And_Configuration_Manual_7.2.pdf”

Find “Chapter 7: Maintaining the CX System, subsection UPDATING USER AGENT FILES” (pp 479 on my copy)

Follow the instructions to update the browscap file and the Event Value lists. The doc is a bit weak, so look at your existing Event Value lists for Browser, BrowserVersion, and Platform (in event editor) until you can recognize the patterns in each file, and then experiment with the UserAgentRevealer.exe settings, so you know what settings produces each browser list. Then export your browser lists. One shortcut the document doesn’t mention; UserAgentRevealer.exe produces CSV files, and the event editor list import feature expects txt files. If you simply rename the .CSV output to .TXT, you can import it to the event list. Event list imports are limited to 1000 items, so you may have to import the txt file once, then wack off the top 1000 items, import it again, repeat, until you get all the items imported. The importer recognizes and ignores duplicates, so this approach is fine.

It is important that you take some time to study each of the browser list (Browser, BrowserVersion, platform) in the event editor lists tab, and take some time to study the various options on the UserAgentRevealer.exe, to be sure you match correctly the CSV output of the UserAgentRevealer.txt to the correct event editor list.

PowerShell function to compare registry keys across multiple computers

 

The Tealeaf registry settings can and should be controlled by the Tealeaf Management Service (TMS). However, I’ve occasionally had some problems with TMS not completing, and I have been left wondering if all the Tealeaf keys, properties, sub-keys, and their properties, are correctly in sync across multiple computers. Until TMS gets a “validate” feature, I’ve developed a PowerShell script that will report on any discrepancies between the Tealeaf keys across multiple computers.

To use this, the Tealeaf computers will need to enable Powershell-Remoting. There are plenty of instructions for this on the Internet, and I plan to record a blog here shortly on how to do that.

Remember to define the $computerNames array. It only makes sense to compare Canisters to Canisters, and HBRs to HBRs, so you will probably need to create two lists of names, and run the Compare-AllRegKey function twice.

Here is the code. Paste it into a PowerShell window, and the Help is included. Enjoy!

<#
.SYNOPSIS 
Compares Registry Key Properties and subkeys across multiple computers
.DESCRIPTION
The function Get-AllRegKey  will recurse down from a given key, returning an array having
    the key's properties, subkeys, and their properties and subkeys.
Provide Get-AllRegKey a list of computernames, and it will remote to those computers
    and return the properties, etc. of the same key on the remote computer
Provide function Compare-AllRegKey with the name of the reference computer, a list of
    two or more computer names, and it will call Get-ALlRegKey to retrieve the key 
    information from all the listed computers, then use Compare-Object to return 
    just the differences.
If you want more control over the Compare-Object step, you should modify the 
    function (suggestions welcome for an efficient/concise way to add Compare-Object 
    parameters to the Compare-AllRegKey function)
  
.PARAMETER ComputerNames
In the Get-AllRegKey function this is a single computer name or an array of computer names
In the Compare-AllRegKey function, this must be an array of at least two computer names
.PARAMETER RegKey
a single registry key/hive from which recursion starts, using the Registry Provider syntax
The value defaults to the current-scoped variable $DefaultRegistryKey
.PARAMETER -ReferenceObject
Applies only to the Compare-AllRegKey function, this parameter identifies the computer
against which the other computers are compared. This string must be one of the computer
names found in the ComputerNames parameter.
.EXAMPLE
C:\PS> Get-AllRegKey
.EXAMPLE
C:\PS> Get-AllRegKey -RegistryKey 'HKLM:\SOFTWARE\Microsoft\PowerShell'
.EXAMPLE
C:\PS> Get-AllRegKey -ComputerNames @('localhost','RemoteCN')
.EXAMPLE
C:\PS> Get-AllRegKey -ComputerNames @('localhost','Computer1','Computer2')
.EXAMPLE
C:\PS> Get-AllRegKey localhost 'HKLM:\SOFTWARE\Microsoft\PowerShell'
.EXAMPLE
C:\PS> Compare-AllRegKey -ComputerNames @('localhost','RemoteCN') 
.EXAMPLE
C:\PS> Compare-AllRegKey -ComputerNames @('CN1','CN2') -ReferenceObject CN2
#>

################################################################################
# Default values for the Registry Key 
$DefaultRegistryKey='HKLM:\SOFTWARE\Wow6432Node\TeaLeaf Technology'

################################################################################
# The scriptblock that does the actual work of creating an object representing a 
#  registry key and all it's properties and subkeys and their properties. No Defaults
#  This is recursive and remoteable
$_getRegKeySB = {Param($RegistryHive)
  # Create a local named function
  function _getRegKey {
    Param($RegistryHive)
    # $data is an array, local to each loop of the recursion
    #    initialize it with the name of the hive/key
    $data=@($RegistryHive)
    # Get the hive/keys properties, excluding the ones added by PS
    $props = Get-ItemProperty -Path $RegistryHive | 
        Select * -Exclude PS*Path,PSChildName,PSDrive,PSProvider
    # if $props is empty, piping it to get-member produces an error message
    # so test it for non-null first
    if ($props) {
      $props = $props | get-member -memberType NoteProperty
      # prepend each property with the full name of the key, and add it to $data
      foreach ($p in $props) {$data+=("$RegistryHive`:"+$p.Definition)}
    }
    # recursivly call the same algorithm for any subkeys of the hive/key
    foreach ($sk in (get-item $RegistryHive).GetSubKeyNames()) {
      # if there are any subkeys, append their data to the current data.
      # Use the full name of the key
      $data += (&_getRegKey (($RegistryHive)+'\'+ $sk))
    }
    # the local named function's output is the array representation of the hive/key
    $data
  }
  # Call the local named function
  &_getRegKey $RegistryHive
}

################################################################################
# Across all computers, get the key and subkeys from the registry
#  returns a hash of array objects, keyed by computer name
function Get-AllRegKey {
  Param (
    # Single computer name or an array of computer names. 
    #  Defaults to the "current scoped variable by the same name"
    $computerNames = $computerNames
    # A valid key
    ,$RegistryKey = $DefaultRegistryKey
  )
  # create the empty hash
  $AllRegKey = @{}
  # iterate over each computer name
  foreach ($cn in $computerNames) {
    switch ($cn) {
      # If the computer name is localhost, or the same name as hostname
      #   use the Call operator to call the scriptblock, and assign the array returned 
      #   to the hash using the current computername as the key
      {$_ -match "localhost|" + (hostname)} {
        $AllRegKey.$cn = &$_getRegKeySB $RegistryKey
        break
      }
      # for all other computer names, execute the command remotely using invoke-command
      default {
        # pass the scriptblock to the remote computers
        #  assign the array returned to the hash using the current computername as the key
        $AllRegKey.$cn = (invoke-command -Scriptblock $_getRegKeySB `
            -ArgumentList $RegistryKey -computername "$cn")
      }
    }
  }
  #return the hash of arrays
  $AllRegKey
}

################################################################################
# Across all computers, get the key and subkeys from the registry
#  returns a hash of array objects, keyed by computer name
function Compare-AllRegKey {
  Param (
   # Must be an array, with 2 or more members; 
   #   defaults to the "current scoped variable by the same name"
    $computerNames = $computerNames
    # A valid key
    ,$RegistryKey = $DefaultRegistryKey
    # The name of the computer to use as the reference 
    ,$ReferenceObject
  )
Begin {
  # If the argument for $ReferenceObject is null, then default 
  #    to the first element of $computerNames
  if (!$ReferenceObject) {$ReferenceObject =$computerNames[0]}
  else {
    # Validate that the $referenceObject is an element of $computerNames
    if (!($computerNames -contains $ReferenceObject)) {
      throw ("{0} is not a member of the list {1}" -f $ReferenceObject, `
         ($computerNames -join ','))}
  }
  # Get the Registry Key data for all computers 
  $AllRegKey = Get-AllRegKey $computerNames $RegistryKey

} 
Process {
  # Iterate over the computernames, excluding the $ReferenceObject
  $diff = @{};
  foreach ($cn in $computerNames | Where {$cn -ne $ReferenceObject}) {
    # compare $ReferenceObject to the remaining objects, accumulate into $diff
    $diff.$cn = Compare-Object -ReferenceObject $AllRegKey.$ReferenceObject `
        $AllRegKey.$cn | Where-Object { `
            ($_.SideIndicator -eq '=>') -or ($_.SideIndicator -eq '<=') }
  }
  # Return the difference hash
  $diff
}
}

# The following lines will compare the default registry key 
#  across the default array of computernames
Compare-AllRegKey 

Technorati Tags: PowerShell,TeaLeaf,TMS,Registry

PowerShell Custom Object Factory

I needed to create custom objects with specific properties for programming against a WPF GUI, because WPF programming is simplest when it binds properties of controls to properties of an object. But I loathed the thought of writing a function to create a custom object, and having to duplicate the parameter name as a Property name in the new-object call. I also wanted to have default values for the object’s Properties.

The following solution is a template – Use it when you need a New-BlahBlahObject function. Simply add the parameters you want to the Param() block, and this function will hand you back a PSObject with the Parameters turned into Properties. As a bonus, the ParameterSetName property allows you to use this as a kind of object factory – depending on which arguments you supply, different Properties will get created, and the value of the ParameterSetName will tell you which properties are present.

You can do a lot more customization before the call to new-object in the last line of the function, by modifying the $p2 string. You can also add functions to the object before you return it, like Add, so you can “add” two of these custom objects together.

[Author’s note: If anybody know how to add a “code” widget to a Blogger Blog, please tell me so in a comment!. The following formatting is terrible… I hope word-wrap doesn’t prevent cut’n’paste from working]

function New-CustomDemoObject {
  [CmdletBinding(DefaultParametersetName='All')]
  Param (
   [parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $Workbook = 'T1.xlsx'
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][switch] $isVisible
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][switch] $isKeepOpen
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][string] $SortDirection = 'Ascending'
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][string[]] $DatabaseConnectionStrings = @('connecstr1','connectstr2')
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $ParameterWithDefaultArray = @('connecstr5','connectstr6')
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][hashtable] $CommandStringsHash  = @{cmd1=@{cmdstr='a command';vo=1;so=2};cmd2=@{cmdstr='b command';vo=2;so=1}}
  ,[parameter(ParameterSetName='PathToDir',ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $PathToDir = 'C:/Logs'
  ,[parameter(ParameterSetName='PathToZip',ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $PathToZip = 'C:/Logs/Backups'
  ,[parameter(ParameterSetName='PathToZip',ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $PathInZip = './'
  ,[parameter(Position=0, ValueFromRemainingArguments=$true)] $RemainingArgs
  )
  # Snippet that creates an object from the parameters
  # Create a list of the common parameters as a RegExp match string
  $cp = iex 'function gcp{[CmdletBinding()]Param()$p1=@();((gci Function:$($pscmdlet.MyInvocation.MyCommand)).Parameters).Keys|%{$p1+=$_};$p1|%{"^$_`$"}};[string]::join("|",(gcp))'
  # Get a list of this function's parameters, eliminate the commonparameters, make a hash with the parameter name and type
  $p1=@{};((gci Function:$($pscmdlet.MyInvocation.MyCommand)).Parameters).Keys|?{$_ -notmatch $cp}|%{$p1.$_=((gci Function:$($pscmdlet.MyInvocation.MyCommand)).Parameters).$_.ParameterType.Name}
  # Iterate this function's real parameters, turn switches to bools, remove ActionPreference (and you can do whatever else you want to in this loop)
  $p2=$p1.Clone();$p1.Keys|%{$key=$_;switch ($p1.$_) {'SwitchParameter' {$p2.$key='Bool';break}'ActionPreference' {$p2.Remove($key);break}}}
  # Add the parameterset as a property (Tells you which parameters are present), and create a new object using the hash of argument names and values
  # This version of the last line creates strongly-typed Properties, but won't handle Parameters/Properties of type arrary or hashtable
  # new-object psObject -Property (&(iex $('{@{ParameterSet=[string]'+ "'$($PsCmdlet.ParameterSetName)';" +[string]::join(";",($p2.Keys|%{("{0}="-f $_+'['+$p2.$_+']'+'"$'+$_ +'"')}))+'}}')))
  # This version of the last line creates untyped Properties, and does handle hashtables and arrays. see also this blog https://jamesone111.wordpress.com/2011/01/11/why-it-is-better-not-to-use-powershell-parameter-validation/
  new-object psObject -Property (&(iex $('{@{ParameterSet=[string]'+"'$($PsCmdlet.ParameterSetName)';"+[string]::join(";",($p2.Keys|%{("{0}="-f $_+"`$$_")}))+'}}')))
}

$newobj1 = New-CustomDemoObject # If none of the specific ParamterSet arguments are supplied, the default ParamterSetName 'All' causes all parameters to be created on the PsObject as empty Properties
$newobj2 = New-CustomDemoObject -Workbook 'another.xlsx' -PathToDir 'D:\Logs'
$newobj3 = New-CustomDemoObject -isVisible -PathToZip 'D:\Logs\Backups'
#$newobj4 = New-CustomDemoObject -Workbook 'another.xlsx' -PathToZip 'D:\Logs\Backups' -PathToDir 'D:\Logs' # Uncomment this - you will see an error about paramterset cannot be resolved

# Type $newobj1 ( and $newobj2, and  $newobj3) at the comamnd prompt to see the objects that got created

Search Server–Main thread proc crashed on cmd …

I was recently asked what these errors mean (you may see them in the PortalStatus report, or the application event logs):

(16:25 Search Server) - TeaLeaf Search Server Ver: 7.2.12.7296 - Main thread proc crashed on cmd /serverstatus. See TLSrchSrv log. 
(16:19 Search Server) - TeaLeaf Search Server Ver: 7.2.12.7296 - Main thread proc crashed on cmd /wordlist. See TLSrchSrv log.
(16:18 Search Server) - TeaLeaf Search Server Ver: 7.2.12.7296 - Main thread proc crashed on cmd /wordlist. See TLSrchSrv log.

That is not an unusual error. Historically, we started seeing the Search Server “crash” about version 4.x. The developers ended up buttressing the SS service with a number of error recover routines. You are seeing one of them here. The SS is the gateway between the Tealeaf software and the CTREE database that stores the persistent session data service. It is also the heart of the communications between the different  servers and routinely sends very large messages around the network. Depending on what kinds of queries are running, the tealeaf user load, etc, the SS may “crash”, but the error recovery routines are really good about getting it right back up.

The SS logs can be opened, and you can look for the crash in the log. It will reference back to the command that caused the crash, and the user/IP address that issued the command. You can take that step if you want more information.  However, I usually ignore these errors unless they repeat multiple times a day spread across many hours. If the errors are systemic, then you should contact Tealeaf support to get to the root cause.

CUI Events and JavaScript Exceptions

What happens when a JavaScript exception occurs on a page hosting the Tealeaf Client UI events?

You may have seen a HTTP header called X-TeaLeaf-Page-Cui-Exceptions in the request when the CUI “phones home” with the request to the Tealeaf tlurl (/TealeafTarget.aspx by default), and wondered what this does. Here’s the answer- This HTTP Header (visible in the Request block) will report if an unhandled exception has occurred in any JavaScript executing on that page in the browser. Here are the details.

1. At the time the Tealeaf CUI setup functions execute on the page, the function TeaLeaf.Event.EventSetup is called. This checks to see if a JavaScript function has been attached to the window.onerr property.
   • If a function is attached to this property, then the CUI considers that the application developers already have an onerror handler, so CUI won’t do anything with it. In this case X-TeaLeaf-Page-Cui-Exceptions will always be zero. (not very interesting!)
   • If there is no function attached to this property, then the CUI will attach it’s own onerror event handler, TeaLeaf.Event.tlErrorHandler
2. This window.onerr is the exception handler of last resort. Every time a JavaScript exception occurs (one that is not caught by an exception handler), then window.onerr is called. Once the TeaLeaf.Event.tlErrorHandler is attached to window.onerr, then the TeaLeaf.Event.tlErrorHandler function is called every time a unhandled JavaScript exception occurs.
3. TeaLeaf.Event.tlErrorHandler function increments an internal exception counter and it creates an entry in the CUI events collection. This entry has the type “INFO” and the subtype “EXCEPTION”, and it has the text of the exception message, the URL of the page on which the exception occurred, and the line number on that page where it occurred (pretty cool!).
4. The tlErrorHandler function then forces the CUI to phone home right away, and passes any existing CUI events that have been queued, along with this exception message.

So, a JavaScript exception forces a call to /TealeafTarget.aspx (or whatever your application has configured for the URL where the CUI phones home), and reports the page URL, the JavaScript exception message, and where it occurred on the page, as well as the running total of JavaScript exceptions that have occurred on the page so far. Note that the exception count is not reset to zero when the CUI phones home, so this field will increment if there are multiple JavaScript errors on the page (and each JavaScript error will cause its very own call to /TealeafTarget.aspx).

Replay rule that stops RTV from trying to contact Omniture servers (or other third-party servers)

 

Here’s a tip – if a page is taking a long time to load, it may be trying to contact a 3rd-party tracking site like Omniture. Look at the “view page load detail”s when the replay view of a page seems to hang. You may see calls to 2o7.net, or other domains that are not part of your domain. These calls are RTV replaying embedded web-bugs – typically 1-pixel transparent gifs that are used to track visitors. Very often, I’ve seen RTV sit and wait a long time for a response – but because RTV is running on your desktop behind your corporate internet access rules, the company network disallows calls to those domains. So RTV just sits there waiting for it’s time-out to expire.

You can avoid this long wait, simply right-click the offending URL in the “view page load details”, and select “Don’t allow RTV to visit this URL…”  When the dialog box pops up, you should use wild cards to replace everything to the right of the domain. [example]. You can also replace part of the host name with a wild card too.

This is also the same method you use if you want to be sure that replaying a session does not affect your Omniture statistics. By telling RTV not to visit that URL, your RTV replays don’t get sent to the 3rd party site – hence they don’t impact the statistics reported.

Persisting Application State in Tealeaf Session Attributes

One big problem with events in the Tealeaf model has been that an event on page N of a user session cannot know about user choices made on previous pages in the session. This post will explore a way around this problem, using Tealeaf session attributes to store state information about the user’s interaction with web application.

Lets start with a hypothetical application, explore the problem, and walk through a solution. The application allows three different ways for a user to search for a group of products: Quick, Guided, and Advanced. At the end of the three search processes, the last page of each process all use the same URL and provide the same response page. Nothing in the final page URL, formfields, or response identifies if the user came down the Quick, Guided, or Advanced search path. So how should we create the “last page” event for the three search processes? Making just one “last page” event will not work well for calculating independent conversion rates for the three processes. We need separate events for AT:S:Q:CompleteV:CP,AT:S:GU:CompleteV:CP,AT:S:A:CompleteV:CP ^[1] For these, we need some way of knowing what choice the user made when they selected the type of search to perform.

Session Attributes store data. If we can store a unique pattern into a session attribute for each search path, then we can test the attribute for each pattern and trigger an event when the attribute contains a recognized pattern.

Caveats first: This solution uses events with the category “Session Attribute Event – Per Page”. Events built from this category are evaluated on every single hit. On sites with large traffic, having a lot of these events can put a strain on the CPU. If the Event Processor process cannot keep up with the demand put on it, the Canisters will start to spool during the busy hours of the day, and the message in the application event logs will be similar to “Number of unprocessed events is greater than 20,000…” (where 20,000 is a limit set in the DecoupleEX session agent of the TealeafCaptureSocket.cfg file).

If it is possible to get the development group to add tags to the response, so that there is a specific pattern in the Response page that identifies both the page and the process, then this Session Attribute events solution will not be necessary - simply create events that recognize the tags found in the Response. Unfortunately, it is sometimes not possible to get tags added, due to time or staff resource constraints, in which case, we need an “all-Tealeaf” solution.

1) Create a Session Attribute, call it AT:Current Search Method:Rsp

2) Identify a unique pattern in each of the three Requests that start each of the search processes. If the application implements the Client UI events, then the ID or Name of the control clicked are likely candidates. Another possibility may be the URL-path, or a URLField, may be unique and sufficient to identify each path.

3) Create three events that trigger when a user REQUESTS the first page of each process. In each event, place the unique pattern identifying each process in the AT:Current Search Method:Rsp attribute. You may use the pattern defined in the event, or you may want to use the Start Tag and End Tag. If the Start/End tag method is used, the string between the two tags will be moved to the session attribute. If you don’t use the Start/End tag method, the Pattern data will be moved to the session attribute. Use the Pattern if the pattern that triggers each event is unique. Start/End tags let you use one pattern that recognizes the kind of search started, but use different part of the same buffer to extract a shorter, uniquely identifying string for the session attribute

4) Create the event AT:S:G:CompleteV:B that recognizes the View of the final search request/result . Make this a building block event – it will fire regardless of the process taken.

5) Create three events AT:S:Q:CompleteV:SAEP:B, ,AT:S:GU:CompleteV:SAEP:B,AT:S:A:CompleteV:SAEP:B   using the “Session Attribute Per Page” event type, specifying the AT:Current Search Method:Rsp attribute. Each of the three events specify one of the three unique patterns. Therefore, only one (or none) of the three events will fire on every hit.

6) Create the three final events AT:S:Q:CompleteV:CP,AT:S:GU:CompleteV:CP,AT:S:A:CompleteV:CP. These are compound events that combine the search global event AT:S:G:CompleteV:B with each of the three events in AT:S:Q:CompleteV:SAEP:B, ,AT:S:GU:CompleteV:SAEP:B,AT:S:A:CompleteV:SAEP:B  

That’s it! You have the three events that clearly identify that the search results were delivered to the client, and which kind of Search process generated the final result view. Now that the “Current Search Process” events exist for each kind of search, you can use them anyplace in the search processes where there is ambiguity in the URL, URLField, or Response, to clearly identify what kind of search process is being used.

Related Posts:

A Naming Convention for Events

Why do I see multiple similar sessions when displaying search results in RTV

This is caused by session fragmentation, combined with Automerge. Session fragmentation occurs in the persistent data storage (the Long Term Canister LTC) because users walk away from their browser for long periods. They may return over an hour later, but have not closed their browser window. After a period of inactivity, Tealeaf is configured to move the user's hits from RAM (Short Term Canister (STC)) over to the LTC. These hits are given a FragmentID (AKA Session ID) – an integer with values that range from “1” up to around 2 billion. Every hit in this fragment also has a copy of the TLTSID and all hits in this fragment have the same TLTSID value. When the user returns after an hour, and starts sending hits again, Tealeaf holds these hits in STC (RAM) with a different FragmentID, but every hit still has the same TLTSID value as the hits of the previous fragment. This may happen four or five times over the course of a work day. As long as the browser remains open, all hits have the same TLTSID value, but different groups of hits will have a different FragmentID. Now when you search the LTC for a session, the system returns to RTV the FragmentID where the search terms match. RTV then asks the Tealeaf system for all of the hits in that fragment, AND if Automerge is ON, RTV asks the Tealeaf system for all of the hits of all the fragments within an 6-hour window on either side of the fragment having the match. In the RTV search results panes, each line is a session that the search found. The left column is the Session Identifier. If Automerge succeeded, the value shown in this column for the session is the 32 character TLTSID, then a dash, then the FragmentID. If Automerge didn't find any other fragments, the value of this column for the session shows only the FragmentID integer. But why does RTV sometimes show two or more identical rows having nearly identical Session Identifier values? If the search matched in two or more fragments of the same session, then Automerge runs for each fragment that matches. If the session lasts long enough, some fragments are outside of the six-hour window on either side of the fragment that matches. So Automerge is going to reassemble different sets of fragments depending on which fragment the search term was found in. You will see a row for each session fragment that matched, and the Session Identifier column for each row will have the same 32 character TLTSID, but different FragmentIDs! The other columns of each row shows how many hits were assembled on each side of the matching fragment, the timestamp of the first fragment, and the duration of the merged fragments. And this is why you may see multiple rows which look a whole lot alike, and when you replay these, you are seeing the same session -just different portions of it, with a lot of overlap between each merged set of fragments.

Another thing you should be aware of regarding FragmentIDs, AKA Session IDs: These integers are assigned on a per-Canister basis, start at “1” with a brand new canister, and simply increment. If your system has multiple Canister servers, these integers will increment at different rates, depending on how the traffic is sent to each Canister. If a new Canister is added, the FragmentID for that new Canister starts at ‘”1”, regardless of the values in any other Canister. Tealeaf Portal and RTV both let you search for a “Session ID”,, which is searching the value of the FragmentID. But I’d suggest you avoid using this search term – you can end up finding totally disparate sessions if the same integer appears in different Canisters for completely unrelated session fragments from different users.