Tealeaf Ecosystem High-Level Block Diagram and Description

The following is a very high level block diagram and short description of ALL the components that make up a full tealeaf system, including the optional components. I’ve written this post as the way I implement systems, and other configurations are possible. For example, PCI blocking can be done at the HBR level or the Processing/Canister level, but doing so means more servers must be made PCI compliant, costing more money and carrying more risk. In another example, I’ve heard that few other implementation consultants bother to implement the Statistics database, but the effort is minimal and the data valuable, so I’ve never understood why not. Since this is a high-level description of the tealeaf components, I’ve chosen to describe them in declarative style without trying to enumerate all the possible options for configuring each component.


[Edit 06/05/2014: A PDF version of this drawing can be found here.]

Below are descriptions of each block.

Network Components

This block encompasses all of the non-tealeaf servers that are involved with the duplication and transmission of the data packets.

Packet Duplication

Any network component (Tap, Switch Span, Load Balancer) that performs the actual duplication of TCP/IP packets. SSL decryption may be performed at this layer, or lower at the Passive Capture Appliances (PCAs) layer

Packet Transmission

The network component (direct-connect crossover cable, switch, Gigamon) that connects the duplicated packet stream to the Passive Capture Appliances (PCAs). These devices may also connect the duplicated packet stream to other devices, such as intrusion-detection systems that need the data.

Passive Capture Appliances (PCAs)

Redundant Linux servers connected to the duplicated packet stream that reassemble the packets into request blocks and response blocks, and these into hits.  SSL decryption may be performed here, or higher at the Packet Duplication layer. PCI data blocking and/or masking should all be performed at this layer. The TLTSID sessionization value will be inserted here if not already present in a HTTP cookie from the request or response block. PCA servers should be treated as PCI critical components. Below this layer, no PCI data is present. PCA servers are managed via SSH and/or a tealeaf web console (GUI) that may only be accessed by tealeaf admins from the Health-Based Routers (HBRs).

Health-Based Routers (HBRs)

Redundant Windows servers connected to the PCAs whose primary purpose is to distribute the traffic stream to multiple Processing/Canister servers. Distribution is session-sticky (using the TLTSID cookie), and normally done with a statistical even-distribution algorithm to send roughly the same number of sessions to each Processing server.
Overall, this diagram is for a production tealeaf system, but a good corporate tealeaf implementation includes both development and QA tealeaf systems as well (much smaller of course). Developers of tealeaf events need a small stream of production data sent to the development tealeaf system in order for developers to have data against which to create new events. The HBR servers include the capability to extract specific sessions or a statistical random fraction of sessions from the production stream and send those sessions to the development tealeaf system, as indicated in the diagram.
The HBR servers monitor all of the Processing/Canister servers, and if any Processing server stops responding, the HBR servers take that Processing server out of rotation and re-distributes the traffic to other servers. If a Processing server comes back alive, the HBR servers begins sending traffic back to that Processing server. Processing servers are cycled and self-checked every night, at different times, and the HBR routers must take each Processing server out of rotation, allow sufficient time for most sessions on that server to end, recognize the Processing server has stopped responding (while it self-checks) and recognizes when it comes back on-line, and begin sending it data again.
HBR servers do no data storage, but operate on the data stream. Examples include robot identification (User-Agent or IP based); deletion of hits based on IP address or URL or any combination request or response patterns; rewrite the Remote IP address using the latest HTTP_X_FORWARDED_FOR value; copying cookie values like a SID to the appdata section for indexing; condensing the referrer domain to provide meaningful referrers; extracting price and currency information from a page; normalizing page URLS to remove locale country codes and many other operations.

Processing/Canister servers

Redundant Windows servers connected to the HBRs that provide the storage location for hits and process these hits looking for patterns. The duration that sessions persist for replay and analysis is a function of traffic density and the amount of disk space the Canister servers provide. Sessions are extracted from the canisters on demand to replay in the Replay Server or RealiTea viewer.
Processing of hits is shorthand verbiage for the complex pattern recognition performed by the tealeaf Events system. Pattern recognition is done against individual hits, against session metadata, and by combinatorial logic looking at all events that have occurred in the entire session. In addition to looking for patterns, match groups can be defined to extract substrings from the data, and the number of times each substring occurs can be recorded. Metadata such as the “time-into” a session that a pattern occurs can be extracted. Extracted data can be grouped into sets. Event processing and data extraction performed in the Processing/Canister servers can be very complex.
If the optional cxConnect Real-time data extractor is installed, the Processing servers create messages for each event configured for real-time extraction, along with information on each hit the event occurs on, and these messages are delivered via a tealeaf pipeline to the cxConnect server.

Reporting Server

Non-redundant Windows server connected to the Processing/Canister servers that poll the Processing servers for their traffic and event counts, and provides the primary web-based GUI interface for users to see reports on this data. Traffic and Event data is collected from the processing servers, and stored in the Tealeaf SQL Reporting database. Dimension aggregates are calculated and stored in the database during each collection run. Daily, Weekly and Monthly aggregates are also calculated periodically. The Reporting GUI provides an interface to create and view reports on this data. Every tealeaf PCA, HBR, and Processing server reports it’s raw traffic statistics to the Reporting server, which stores this information in the Tealeaf SQL Statistical database, and the information is available in the Reporting GUI.

Tealeaf SQL Database server

Non-redundant Windows server running Microsoft SQL Server, or a corporate server running Microsoft SQL server. Tealeaf creates and uses three SQL databases (System, Reporting, and Statistics). The SQL server is usually managed by the corporate DBA team, and not the tealeaf admins. The tealeaf admin team works closely with the DBAs to install, update, maintain, and backup these databases.

Replay Server

Usually a service running on the reporting Server, the Replay server may be configured as a stand-alone Windows server. The Replay server is used to create visual replay of user sessions from the data stream of hits (request/response pairs) stored in the Processing/Canister servers. The Replay is web-based, and does not require any executable to be installed on a tealeaf user’s computer. See also the RealiTea Viewer section for an alternative method of replay.

cxOverstat

A feature that can be turned on that allows the Replay server to display the heatmaps and other functions provided by the cxOverstat feature.

Archive Servers

Optional Windows server(s) whose purpose is to extract a subset of sessions from all Canister servers that meet a selection criteria (most often, a purchase session or a trade session), and store these session for a period of much greater duration that the Canister servers. For example, if Canister servers are ALL storing sessions for 30 days, the Archive servers may store purchase sessions for two years. These can also be configured as “non-tamperable”, which provides a hash-based mechanism to prove that a session replayed from the Archive server is the same session hat was originally captured from  the web site.

TLI servers

Optional Windows server(s) that store and make available for replay certain static content of the web site. During replay, the images and JS files are loaded from the web site. With a TLI server, these files are stored each day, and during replay of a session for say, three weeks ago, the images and JS files as they were three weeks ago, are used in the replay. This provides better fidelity of replay that is closer to the actual historical session. The drawback is the amount of storage needed to keep historical static content.

cxConnect Servers

Optional Windows server(s) whose purpose is to provide the interface that extracts data from the tealeaf Processing/Canister servers and makes that data available to external systems. There are two separate data feeds available. The real-time data feed is a set of (configurable) selected event messages with parameters that are sent by the Processing/Canister servers to the cxConnect servers using a tealeaf transport pipeline. The cxConnect servers have two distribution choices for this data – log to file and/or send to to a TCP/IP listener on an external system. The real-time feed is most often used to feed a Complex Event Processing (CEP) listener system. In turn, these systems drive real-time decisioning systems that modify the web application’s responses to the user based on their past actions in the session. The other data feed available from the cxConnect systems is a scheduled (typically hourly or daily) batch extract of detailed information regarding users, sessions, hits, parameters, events, and dimensions. The information is stored in flat files that conform to the Microsoft SQL Bulk Load (BCP) format. tealeaf provides an example schema for creating a relational SQL database, and script jobs for loading the corresponding BCP files into these tables. This is the reference mechanism provided by tealeaf for putting the data into a fully relational database.
The following three pieces of tealeaf code are implemented on the web sites and native mobile applications

Cookie Injector

Very small piece of code running on the web servers that adds the three tealeaf cookies. The TLTSID is a non-persistent cookie whose value does not change as long as the browser window remains open. The TLTHID cookie is a unique identifier assigned to each hit. The TLTUID is a persistent cookie left on the browser, whose value is sent each time the user revisits the site. All three cookies are 32-character GUIDs.

Client-Side Recording (CUI/SDK) Library

JavaScript (JS) library added to the web site and called from the web pages. This library implements the recording of the DOM events on the web page, and transmits the page’s DOM event information (mouse actions, keyboard actions, page rendering time, etc.) back to the tealeaf system. Includes a target page to be added to the web site that the library will call back to. The CUI/SDK is a very important piece to implement properly for sites that use anything similar to the “one-page” technique. The implementation of the CUI/SDK provides a much higher fidelity replay of the session.

Mobile Library

JavaScript (JS) library added to the Mobile web site and to the mobile device native application (IOS or Android) called by the device.This library implements the recording of the user’s interactions with the device and the page. Screen click, key-press, swipe, pinch, rotation and other page interactions are recorded and transmitted back to the tealeaf system. The implementation of the CUI/SDK provides a much higher fidelity replay of the session.
The following executable is installed onto the tealeaf user’s computer for replay of sessions

RealiTea Viewer

This is an executable program that can be installed onto a tealeaf user’s computer to allow for replay of user sessions. In addition to just replay, it allows multiple sessions to be downloaded to the user’s computer, and provides searching and analysis capabilities for patterns across these downloaded sessions. It includes the ability to customize the view panes and data fields displayed for both hits and for session metadata.
The following systems are not part of the tealeaf ecosystem, but if they exist in the company, they may be fed with data from the optional cxConnect servers.

SQL Relational Database Server for tealeaf events

Non-redundant Windows server running Microsoft SQL Server, or a corporate server running Microsoft SQL server. Tealeaf provides a reference schema for a relational database that links together sessions, hits, query parameters, events, and dimensions. The cxConnect data extractor populates these tables. These tables provide a very rich source for analytics against the user behaviors.

Real-Time decisioning systems

A system that modifies the web page contents based on the user’s past actions. Usually some kind of Complex Event Processing system tied into the web servers.
The following software constructs are resident in the corporate Active Directory structures.

Active Directory Security groups for tealeaf

At a minimum there are two AD security groups. One enumerates the userids which are allowed to use the tealeaf reporting GUI and allowed to access the session data stored in the Processing/Canister servers for replay. The other enumerates the userids given access to tealeaf at an administrative level. These may be Global groups or Domain-local, meaning that userids from multiple forests are supported if desired. Additional AD security groups may be created for teams such as fraud investigation teams, which are given access to encrypted PCI data, should the system be configured to encrypt certain fields instead of blocking them. Only member of these specific AD groups will be able to see the PCI data in clear text. Normal tealeaf users and tealeaf administrators see encrypted gibberish. PCI fields that are blocked instead of encrypted are always replaced with ‘X’ for all users.

Conclusion

I hope this overview is useful in understanding at a very high level all the components of the tealeaf ecosystem. As with all large complex computer systems, the features and components are evolving, so this post will eventually become obsolete. However, as of tealeaf version 8.8 in the spring of 2014, this should be a pretty complete picture
Feedback, comments, and questions are always welcome. Happy Tealeaf-ing!

Tealeaf Dimensions

Lets explore some usual and unusual ways to use dimensions in tealeaf reports. I’ll assume you know the basics of what dimensions are, and the standard way of using them. To review quickly, and event’s value can be stored in a dimension. When you look at a report for that event, in a hour’s period, you can see how many times the event fired (the event’s count) for that hour, or, you can look at how many times each value occurred in that hour. Of course, you can look at more than an hour – but lets keep the verbiage and phrasing simple. When you setup the dimension, you can tell the dimension to record every value the event ever has, or, you can create a whitelist – a list of values that you want to see counted. There are blacklists too, but even after 14 years, I’ve very seldom found a use for those.

Now lets digress just a bit, and explore what dimensions and whitelists do not do. A value in a whitelist is not an event condition. You cannot say “trigger this event only if a whitelist value is present”. Nope. If you have three values in the whitelist, and the event fire 1000 times, but none of the event’s values are in the whitelist, then the report will show the event occurred 1000 times, and the report will list ‘other’ as the sole dimension value. So you see, the whitelist does not prevent the event from firing – it just identifies the specific values you want to see details about.

Another thing dimensions cannot do (at least, as of release 8.7x) is trigger an alert. Often we want to trigger an alert if a specific value of an event occurs, or if an event’s value matches one of a list of such values. Alas, it just doesn’t work that way. An Alert gets triggered based on the event’s count, not it’s whitelist’s value’s count. So,you cannot use a whitelist to trigger an alert. Hope this tidbit saves somebody a wasted day trying to make that work.

As an aside – if you need to create an alert for specific event values; 1) create a building-block event, setting this event’s value to whatever you are looking for. 2) Create a second event that triggers only when the first event fires. Put your list here, and compare the list to the first event’s value. If the first event’s value matches the list, the second event will fire. Use the second event to drive the alert. Therefore, the alert is based on the second event, and the second event only fires if the value of whatever you are looking for matches the list. If your conditions are complex (an IP in the range of blah to bal; OR a userid of ‘johnDoe’; OR a useragent string containing ‘Snap!’) this technique makes it pretty easy. Individual Building Block events whose values are IP, LoginId, and useragent, then a single event that looks at each Building Block event, has the specific list of interesting values for each, and combines results of the list comparasion according to your needs to determine if the event fires. We do this for ‘Session of Interest’ Alert, which goes to security teams (in real time) whenever a specific set of conditions occur. Then they can go watch that session. If enough folks leave a comment expressing an interest, I’ll post example code for the list matching.

Back to the main thread <grin>

Dimension value counts have to be recorded, hence events with dimensions contribute to the size of the database aggregation tables. Looking at the Tealeaf DB statistics page in the Tealeaf portal’s administration view, lots of tables end in _AGG. If any get over about 40M rows, you are going to see performance degradation in the Data Collector’s speed, in report generation, and surprisingly, in the time it takes the event editor to commit new event definitions. Get too many events with large numbers of unique dimension values, and you’ll be unpleasantly surprised to find it takes fifteen minutes to “save” a new event.

But dimensions can be put to great use with a little care. Anything with a small number of unique values are perfect candidates. If you are interested in the viewing choices your customers make for a particular product, and want to see their choices broken down by product size or color, use a dimension for size, and another for color. Fire the event on a specific product SKU, record the customer’s size and color choice in a report group associated with the event, and you can see a report of how often the product was viewed in each size and color combination. Have a number of similar products? Create the event to fire on any of a list of product SKUs, and, add the product SKU as a dimension in the report group. Now your report will show multiple SKUs, sizes and colors. Just don’t go overboard – recording too many different SKUs in this one event will lead to aggregation table bloat, again.

Another fun thing I like to do with with Dimension Groups is to create histogram reports.You know dimensions and whitelists let you record unique values and dimension groups let you group these values. One good example is to use the Session Duration session variable. On each hit, the system updates this session variable with the the current duration of the session. You can create an event that records this. Lets say you create an event on the “purchase complete” page, and record the value of the Session Attribute “Session Length Running Time (sec)”. Now you have, for each purchaser, how long it took for them to get from start to purchase. Now lets group these counts into buckets. Create a Dimension Group, and each element is 0-1799;1800-3599;3600-5399,… etc. Populate this Dimension group with the Session Attribute “Session Length Running Time (sec)”. The internal name of this attribute is S_TOTAL_TIME. Assign this Dimension group to a Report group and the Report Group to the event. The resulting report shows 1) how many purchases in the hour; 2) how many customer made their purchase in 0-3 minutes; 3-5.9 minutes; 5-8.9 minutes; I find this technique to be interesting – it helps the business visualize how long it takes most customers to make a purchase. You can extend it so the purchase event triggers for a specific product – and then compare the two reports to see if different products take differing time to purchase. I wish there was some way to put the comparison up on a single report, but I haven’t found a way to do that.

One more I have not tried personally, but would be interesting… The Report Group above defines how long it takes to reach a purchase from the start of the session. What if you want to know how long it takes to get from Event A to Event B, and report that as a histogram? For example – start of checkout to purchase complete? Go create Events A and B for Start of Checkout-BB and End of Checkout-BB, and record the Session Length Running Time (sec) as their values. Create an Event C “Time in Funnel (sec)”, that fires when both events have fired. Edit it in Advanced mode, and record the value of this event as the difference between the value of Event B – the value of Event A. Note that you may need to account for multiple event A’s occurring before the Event B, so make sure you use the latest (highest numbered) event A in the event A collection. Now use the value of Event C to populate a Dimension Group along the lines of the last example, and use the Dimension Group to populate a Report Group assigned to Event C. Report on Event C, and the report showing how many users spent how many seconds in the checkout funnel is available. I have not tried this one yet – please let me know if you encounter any difficulties with this one.

I hope these work for you, and spark some more interesting ideas on using dimensions!

The ErrorCode and ErrorMessage Events

A really useful group of events are ones that can detect the error messages displayed on your site. But these can be difficult to get right.  Lets go through creating events to detect error messages, and some pitfalls to watch out for.

What’s the difference between an error message and an error code? In some sites, no difference at all – all you have to work with is a message string the user sees. But if you have a site that supports multiple languages, you may have the same error message presented in two or more language-specific strings. If you have this situation, you are going to want to know not just the specific string being shown, but how often each error condition displayed regardless of the language. For this, you will need for the site developers to insert an error code (it can be "non-displayable” in the page), as well as the error message.

Here’s an example: class="fError">! There are no ...<!--ErrCode:FA65—></

After you’ve isolated the error string from it’s surroundings, to break that string into the language-specific error message as well as the error code requires only two simple regular expressions:

$REMessage = /(.*?)<--/;
$RECode = /<!--ErrCode:(.*?)-/;

Getting the error string separated from its surroundings, though, that may be tricky…

First, Last, Count, and Aggregate Field Errors

If the site has only one error message on a page, a simple hit attribute and pattern is all you need, but in practice it is seldom that simple. Most often, the site may have multiple error messages on a single page. In order to understand which error messages are most common, we need to extract every error message.Hit attributes combined with Basic mode events can record the first error, or the last error, or the number of errors on a single page, but there is no provision in tealeaf for an event to record a collection of errors. The best we can do is create an event that will aggregate all of the errors into one string. We are going to need an advanced mode event to do the aggregation.

Visible and Invisible Errors

The event to aggregate the errors would be much easier if all error messages were visible, but alas that is not always so. In many sites, the developers will deliver all the error messages as part of the HTML of the page, but will use a CSS display attribute to control the visibility of the message. So we need to further refine our event so it only records Visible errors. Here are examples that show the same error message in multiple visible and invisible forms.

Visible:

class="fError">! There are no ...<!--ErrCode:FA65--></div>
class="fError"><br />! Make a selection for ...<!--ErrCode:TI1--></span>
class="fError">! Please enter a valid...<!--ErrCode:A21--><BR></span>
class="fError">! Please select a proper option.</span></p> – Note that the developers left out the error Code!
class="fError" style="color:#CC0000;">! Please enter a New ...<!--ErrCode:V1--><br/></span>

Invisible:

_CustomValidator1" class="fError" style="color:#CC0000;display:none;"></span>
<p class="fError" visible="False"></p> -- Note that this doesn't even have the trailing <span>
class="fError">< – This is an “empty” error message

Ignored: (our business users have told us don’t want to include this error when analyzing message)
class="fError" style="font-size:80%">! Your membership is expired.</div><

Regular Expression Patterns

The following Regular Expressions (RE) will capture each type of field error. There is some overlap -  patterns which are both invisible and empty will fire both of their respective REs. Instead of writing down what each pattern is doing, let me send you to a on-line tool for making REs that will do a great job – just paste the RE into the online tool, check ‘Explain’ and hit submit. I’ve looked at dozens of RE test tools – this is my personal favorite, and the one I use when developing advanced mode events: http://myregextester.com/index.php

$REVisible = /class="fError(?:" style="color:#CC0000;){0,1}">(.+)<\/(?:span|div)/;
$REInvisible = /(?:class="fError" style="color:#CC0000;display:none;"|class="fError" visible="False">)/;
$REEmpty = /class="FError"><\/(?:span||div)/;
$REIgnored = /class="FError" style="font-size:80%">/;

Known Error Patterns

Grouping together the known, invisible, empty, and ignored patterns into one pattern “Known”:

$REknown = /class="FError"(?: style="color:#CC0000;display:none;">| style="color:#CC0000;">| visible="False">| style="font-size:80%">|><\/span|><\.+/span|><\.+/div)/;

Event Coverage

Because of the complexity, we’ll also want to have some code to check our event coverage – are we counting every visible error, and ignoring every invisible error? are there any errors that don’t fit our definition of either visible or invisible or ignored?

We can do this on every page with the following events

• [BB] count every occurrence of “class=”ferror” (and alternate formulation with one apostrophe, and no apostrophe or double-quote characters, and zero or multiple white-space characters.
  • /class\s*=\s*([‘”])ferror\1\s/ig
• [BB] count every occurrence of Known Field Errors
• [E] count every occurrence of unknown field errors on a page (total – visible – invisible)

 

Events and Dimensions for Error Codes and Messages.

Finally! We get to the actual events, dimensions, and report groups. In the interest of brevity, I’ll present the Aggregate Visual Field Error Code information. If you want the details on First Visible Field Error and Last Visible Field Error on a page, you can copy/modify these. I will show you the “guts” of the javascript in the advanced mode event for each of these, as well.

All of these events depend on the triggering event G:Err:NumberOfVisibleFieldErrorCode:E When the event G:Err:NumberOfVisibleFieldErrorCode:E appears on a hit, then each of the following events are evaluated. In this event, we set the conditions “statuscode = 200, URL does not include tealeaftarget”, and whatever other conditions you may want to exclude (for example,excluding a specific domain).

In order to search repeatedly through a response, we need a hit attribute (HI) that will extract the response; G:ResponseBody with start tag of <html and end tag </html. In each of the following events, we use the HI to extract the entire response body into a local variable. When Regular Expressions are applied repeatedly to a string, there are internal pointers that remember how much of the string was parsed. If we tried to call the HI function for the response body in each loop , those pointers get reset. So we make a local copy, and loop the regular expression against the local copy.

Events:

G:Err:AggregateVisibleFieldErrorCodeOnPage:BB – [ADV] string consisting of all the visible field-level error codes on a page. Iterate over the response body for the REVisible pattern. Each time a match occurs, extract from the match string the error code. Accumulate the multiple error codes with ‘:::’ as a separator. Store this string as the value of the event.

function YOURNAMESPACE$E__G_ERR_AGGREGATEVISIBLEFIELDERRORCODEONPAGE_E__634853855183542782()
{
    if ($F.getLastFact("YOURNAMESPACE.F_E__G_ERR_NUMBEROFVISIBLEFIELDERRORCODEONPAGE_E__634761380708236960").HitNumber == $H.HitNumber)
    {
        $REVisible = /class="fError(?:" style="color:#CC0000;){0,1}">(.+)<\/(span|div)/ig;
        var $str = $P["YOURNAMESPACEL.P__G_RESPONSEBODY__634753590986008896"].firstValue();
        var $cnt = 0;
        var $resultstr = "";
        while ($matches = $REVisible.exec($str)) {
            if ($matches != null) {
                if ($matches[1] != null) {
                    $RECode = /<!--ErrCode:(.*?)-/;
                    var $RECodeMatch = $RECode.exec($matches[1]);
                    if ($RECodeMatch != null) {
                        if ($RECodeMatch[1] != null) {
                            $cnt++;
                            $resultstr += $RECodeMatch[1]; $resultstr += ":::";
                        }
                    }
                }
            }
        }
        if ($cnt > 0) {
            // Set fact for Report Group: No Dimension Report Group
            $F.setFact("YOURNAMESPACE.F_E__G_ERR_AGGREGATEVISIBLEFIELDERRORCODEONPAGE_E__634853855183542782", $resultstr);
        }
    }
}

G:Err:FirstVisibleFieldErrorCodeOnPage:BB Iterate over the response body for the VisibleRE pattern. Each time a match occurs, do nothing until the last match. Then split the match string into message and code. Store the code string as the value of the event

G:Err:FirstVisibleFieldErrorCodeOnPage:BB Test the VisibleRE pattern against the response body. Take just the first match. Split the match string into message and code.  Store the code string as the value of the event

$REVisible = /class="fError(?:" style="color:#CC0000;){0,1}">(.+)<\/(span|div)/ig;
var $matches = $REVisible.exec($P["UNITEDCONTINENTAL.P__G_RESPONSEBODY__634753590986008896"].firstValue());
if ($matches != null) {
    if ($matches[1] != null) {
        $RECode = /<!--ErrCode:(.*?)-/;
        var $RECodeMatch = $RECode.exec($matches[1]);
        if ($RECodeMatch != null) {
            if ($RECodeMatch[1] != null) {
                // Set fact for Report Group: No Dimension Report Group
                $F.setFact("UNITEDCONTINENTAL.F_E__G_ERR_FIRSTVISIBLEFIELDERRORCODEONPAGE_BB__634761678018624416", $RECodeMatch[1]);
            }
        }
    }
} 

G:Err:AnyVisibleFieldErrorOnPage:E Is the reportable event we look at to see what errors are happening on a page. This event is a simple “OR” of the six BB events, so that, if the first/last/aggregate Visible Error Code/Message events fire, any of them, the Report Groups on this event will record the six values, and the URL (page) on which they occurred.

Note that recording all six dimension values may be overkill for your analytics group. I like to record all six for a few days, and review them with the business information consumers. If they agree that the aggregate dimensions are all they need, it’s easy to remove the unneeded dimensions. You  will also want to be wary of the Message dimensions. While these are certainly more consumer-friendly than the error codes, storage for long strings of aggregated error messages may prove to be a concern. If you decide to record this information, keep an eye on the aggregate table usage over a few weeks, and plan to re-evaluate its usefulness and it’s storage cost.

The Results

After putting these together we can analyze the errors that happen on a specific page. We do this by looking at the event G:Err:AnyVisibleFieldErrorOnPage:E, and homing in on specific URL or groups of URLs, and looking at the error dimensions. When this event fires, we are recording (and viewing) the errors that appear on that page.

On what pages do field errors occur the most often?

On any specific URL, what are the most common visible field errors (aggregate of all errors on the page)?

What are the most common “last seen” error codes when a user abandons the checkout process

The truly actionable information you can glean is “last seen error when purchase is abandoned”. This is the favorite information for a lot of data consumers. Understanding this helps understand what it is about the site that most often impedes the purchase process.

This works because we defined our dimension to record the “lastest” value of a visible error event. We define an event that fires at the end of a session if we detect that abandonment took place (e.g. FPR:Abandoned:S, which is defined as FPR:ReviewRevenueR:S AND NOT  FPR:ConfirmRevenueV:S), and we assign our six error dimensions to report groups assigned to this event. We don’t need the URL as part of this Report group, because as a end-of-session event, the URL dimension would haphazardly record whatever URL happened to be the last one in the session (and most often that would be the tealeaftarget which occurs when the uiSDK phones home the information for the last page). However, the “lastest” error messages may not be the error messages on the last page of the checkout process abandonment, if the user goes and visits other parts of the site, like searches, after their last checkout process page.

So this event is the closest we can get using the built-in tealeaf reporting tools. To get closer to the truth, a better approach would be to use cxConnect to record the error event in an external BI data warehouse, and use analysis queries to ask “for every abandoned sessions, which was the last checkout process page seen, and what were the errors shown”.

Within the limits of the tealeaf reports, here are the “last seen visible error codes when purchase is abandoned”.

We’ve filtered to eliminate the [Null} value. If a checkout process is abandoned, and there are no visible field errors in the session, then [Null] is recorded for this dimensions. In this specific site, many automated tools get to the first page of the checkout processes, producing a “Null” entry here when the automata “abandons” its checkout. An external BI data warehouse would allow better refining these numbers to eliminate robots.

Good luck error hunting! I hope this piece will help you better understand some of the event and dimension concepts in Tealeaf V8.

V8 Process Event Guidelines

Every e-commerce web site has one or more “processes” – sequential steps the user has to go through, to accomplish a goal. The most important process for most e-businesses is the “checkout” process (aka the “purchase” process). In this post, we will create discuss how to create events that track processes, and discuss the characteristics you should build into these events.

At the fundamental level, a client sends a request, and the server sends back a response (view). You should create two process events for each step, one to recognize and count the request, and another that recognizes the response.  When tealeaf recognizes a request event, you know the user actively asked for the next step of the process, and when tealeaf recognizes the response, you know the system presented the information to the user. In my post V8 Event Naming Conventions I discuss how to create event names that reflect this difference. The request event can be based on the URL of the page, and the view event  is best based on a unique pattern present in the response when the page is successfully rendered.

However, before we go much further, there are major differences between Apache/IIS/ColdFusion server technology when it comes to how the technology uses the URL, and this difference makes a huge difference in how the Request for a step is recognized. <Insert moew on rthis>

Now that we are all using the same event names to mean the same logical view of the steps regardless of our site’s technology, onward!

Conceptually, a simple checkout process is “Cart View”, “Billing Info”, “Final review” and Confirmation. This produces and eight step checkout processes

Process Step Events for Every Step

The eight steps that measure every step occurrence are P:CartViewR:E, P:CartViewV:E, P:BillingInfoR:E, P:BillingInfoV:E, P:FinalReviewR:E, P:FinalReviewV:E, P:ConfirmationR:E, and P:ConfirmationV:E. When we create the events, base the R events on the URL of the page, and V events on the page title or some other string in the response that positively and uniquely identifies the page. Set the event type as ‘count”, evaluate on every hit, and do not add any report group yet.

These events tell us how many times each process step occurs, even if they occur multiple times in a session.

Session Events for Each Step of the Process

The eight Session steps that measure “the session had one or more occurrences of a process step” are named P:CartViewR:S, P:CartViewV:S, P:BillingInfoR:S, P:BillingInfoV:S, P:FinalReviewR:S, P:FinalReviewV:S, P:ConfirmationR:S, and P:ConfirmationV:S. Create each of these based on the presence of the corresponding :E event being in the session. Set the event type as ‘count”, evaluate at the end of the session, and do not add any report group yet.

These events tell us how many visitors got to each step of the checkout process during their sessions.

Pyramiding the Session Events

Now string the process steps together by creating the following events. It’s easy to see why they are called pyramiding events. Event type is count, evaluate at the end of the session, and no report group. These tell us how many visitors saw each step, and did not miss a step. They are the most accurate source of information for conversion ratios, and will be used for error detection as well, as will be shown later.

P:Step1R:S - P:CartViewR:S

P:Step1R&1V:S - P:CartViewR:S AND P:CartViewV:S

P:Step1R&1V&2R:S - P:CartViewR:S AND P:CartViewV:S AND P:BillingInfoR:S

P:Step1R&1V&2R&2V:S -

P:Step1R&1V&2R&2V&3R:S -

P:Step1R&1V&2R&2V&3R&3V:S -

P:Step1R&1V&2R&2V&3R&3V:S -

P:Step1R&1V&2R&2V&3R&3V&4R:S -

P:Step1R&1V&2R&2V&3R&3V&4R&4V:S - P:CartViewR:S AND P:CartViewV:S AND P:BillingInfoR:S AND P:BillingInfoV:S AND P:FinalReviewR:S AND P:FinalReviewV:S AND P:ConfirmationR:S AND  P:ConfirmationV:S

Simple Count and Conversion Ratio Reports:

It is straightforward to create reports that show the counts for each event, and the ratio between any two of them. You can generate step-step ratios, or ratios from first step to any subsequent step, including confirmation.

In the chart examples throughout this post, the site had a seven-step process, so the pictures do not “quite” match up to the events described in the post. I wanted to use a very short hypothetical process for the text, to keep the post focused on core concepts. I trust you will not have any problem making the correlation and bridging the discrepancies between the charts and the text.

Pyramid Event Counts

Pyramid Event Ratios

 

Dimensions and Report Groups

How can we “slice” these numbers to get focused information? By using Dimensions and Report Groups! Lets examine the possibilities.

Things that are unchanged throughout the session (on the site I’m using for this post) include the AKAMI Country Code and the Session’s First Hit Referrer Domain

Things that may change at any time in the session include the Point Of Sale (POS) which the user can control (it changes the language) and is found in each response, the Currency of the purchase (which is based on the country in which the credit card is issued), and is always present in the CartView response, and the AmountList, which is a grouping, or “bucketing” of the cart’s sale amount. Another item that changes (but usually only once) is the customer’s loyalty program level if they sign-in.

Now, if we want to know the event counts for each country as reported by Akamai, we add a report group to each event that consists of a dimension that is populated by an event that fires on the first hit of the session and whose value is the value of the Akamai Country Code in a http header. Then in the report builder, we can drag the FirstSeenAkamaiCountryCode:E-F dimension to the chart of event counts, and filter on the top five. This groups the events by the country code that was seen on the first request of the session, and tells us which country has the highest number of visits to each step. Do this on the chart of conversion ratios, and it tells us which country has the highest (or lowest) conversion ratios

Now, if we want to know the event counts for referring domain, we add a report group to each event that consists of a dimension that is populated by an event that fires on the first hit of the session and whose value is the value of the domain portion of the HTTP_REFERRER. Then in the report builder, we can drag the DomainSessionReferrer:E-F dimension to the chart of event counts, and filter on the top five. This groups the events by the domain of the referrer that was seen on the first request of the session, and tells us which referrer has the highest number of visits to each step. Do this on the chart of conversion ratios, and it tells us which referrer has the highest (or lowest) conversion ratios.

If we want to know the event counts by POS, we add a report group to each event that consists of a dimension that is populated by an event that extracts (from every hit) the POS from the response, and in the report builder we can drag the POS:E-L dimension to the chart, and filter on the top five. This groups the step events by the POS that was last seen in the session. It is important to note that the POS can change after the last step of the process. The pyramid events are session events, and are evaluated at the end of the session. The value of POS stored in the report group for the pyramid events will be whatever POS was last seen in the last hit of the session.

Two-Dimensional Report Group

When the site is multi-currency, you cannot add the amounts of orders together. Adding Yen to Yuan to Reals to Peso to dollars makes no sense. It is important to consider both the amount and the currency together as a two-part object. We can accomplish this with a report group having two dimensions, Currency:E-L and AmountList:E-L. Populating these dimensions can be done with an advanced mode event. The details are beyond the scope of this already-too-long post, but can be found here. <insert>

Now in the report builder we can drag the Currency:E-L and the AmountList:E-L dimensions to the chart. We lose the graphical representation, but the tabular data shows which currency and amount buckets are the best performing. (I have no idea why there are [Null] entries – looks like I’ve got some troubleshooting to do :-)

Multi-dimension Report Groups

For fine-grained analysis, create report groups of multiple dimensions (up to four dimensions). For example, a report group consisting of the dimensions POS:E-L/LoyaltyStatus:E-L/DomainSessionReferrer:E-F lets us investigate the relationships between these three constraints on the purchase process events.

<insert>

Other dimensions you may want to consider adding are TrafficTypeE-F,BrowserType:-F,BrowserVersion:E-F,SessionDurationList:H

Make sure you look at the TrafficType:E-F dimensions or via searching, to see if there are robots getting to the first part of your process

What is “abandonment”

Abandonment is simply any session that has the first step of the checkout process,and not the last step. Use these two conditions to create a simple session event for P:Abandonment:S Since this means our customer left empty-handed, companies want to spend a fair amount of resources to investigate why this happens.What companies need are a way to quantify how much revenue is lost when an abandonment occurs. For this example, we will focus on the amount (and currency) that was in the last P:CartViewV:E of the session.

Extracting “loss” on an abandonment

Scraping off the amount and the currency from a response can be tough. The details are here <insert xref>.To summarize you need two advanced mode events (BB) with three regular expressions (RE). Both events use one same RE to recognize the entire class element that encloses the the currency and amount substrings. Each BB event uses one of the other two REs, to split the substrings out of the class element string. If you are lucky, the developers have repeated that pattern everywhere. If not, you may be looking at multiple Regular Expressions in these events.

Two dimensions are populated by these BB events StoreLastSeenRevenueCurrency:BB and StoreLastSeenRevenueAmount:BB, called RevenueCurrencyList:E-L and RevenueAmountList:E-L. The dimensions are populated with the “Last” value of the BB event, which itself fires on each occurrence of the event P:CartViewV:E.

The two dimensions can be combined into a single report group and attached to any checkout process event, but wait…. lets not waste a multi-dimension report group. The final report group we attach to abandon will have more dimensions…

The amount and currency that is being abandoned…

Extracting Error Messages

See <insertt xref here> for details. To populate the two dimensions x and Y, It takes two BB events with multiple Regular Expressions. These BB events look at a page for any class, span , or div pattern that contains the error class element, extract either the language-specific error message or the language independent error code, repeats down the whole page view, and aggregates these one or more substrings into one large aggregate string.

You can attach these dimensions to a step event, but they will always have the last seen event value – even if the last seen error was two hits back. Instead, we create an event G:AnyVisibleFieldError:E, and attach the two dimensions to this event along with the URL (Normalized) dimension. This provides an event whose dimensions can tell us “what is the most common error” for any specific URL or groups of URLs. That’s pretty potent! If you attached just these to the P:Abandoned:S event, you would know what error messages were seen last most often when visitors abandoned. Again, lets not waste a multi-dimension report group..

Top Error messages…

To find out which error message was last seen most often and sort by revenue lost, create a report group with all four dimensionss we’ve discussed

a/b/c/d

Now chart the P:Abandoned:E event in the report builder to see the culmination of our efforts which error messages are seen for the largest amount of abandoned shopping carts, grouped by currency and amount list and errormessage (and errorcode)…

Throw in the revenue amount list to group these

 

The P:Abandoned Event

• Evaluate at End of Session
• Fires if teh session has a ProcessStart and NOT a ProcessEnd event
• Numeric Type of value
• Store the value amount shown at ProcessStart
• Report group contains RevenueCurency:E-L; RevenueAmount:E-L and AggregateVisibleFieldErrorMessage:E-L

I hope this has helped you understand process events, and given you some ideas for expanding your use of tehm.

V8 Event Naming Conventions

With the introduction of the new event models in Version 8 of Tealeaf, new capabilities  and terminology require another look at how events are named. Event naming is strictly “by convention” – you can name an event anything you want, but after you have 500+ events, finding one again a few months later is much easier if the event names all follow some conventions.
Good conventions means that anybody who looks at an event name can get a very good idea of what it does. Good conventions reduce maintenance cost by reducing the number of duplicated events created by different event administrators, and good conventions make it easy to pass along the institutional knowledge when new event administrators are brought on board.  

Terminology and Definitions

All licensed users have access to the online documentation appropriate to their version of tealeaf. One very useful piece of this documentation is the Glossary of Terms. Everyone in your organization who uses tealeaf or consumes the reports generated by  tealeaf should review this document so that everyone shares the same terminology. To access the Glossary, Start the Portal, enter “Glossary” in the Search Online Help and search. While the results page is empty (as of V8.45), enter Glossary again in “Site Search All Spaces”, search, and you will be shown links to the tealeaf glossary.
The important terms, from an event perspective, are:
Hit Attribute: The pattern you are looking for! This is the foundation of the whole eventing system. Three important things:
1) Text String: - a fixed constant pattern of characters, in either the Request or the Response
2) Start Tag/End Tag – a constant pattern of characters that make up the start of the pattern (Start Tag), and another constant pattern of characters that make up the end of the pattern (End Tag). Everything between the Start Tag and End Tag make up the '”value” of that particular Hit Attribute
3) RegEx Pattern – a way to better refine the data between the Start/End tags. Using RegEx (or RegExpr) patterns and match groups, you can pull out just specific portions of the character strings between the Start and End tags, and give the Hit Attribute the value of just some substring(s) of the full data between the Start/End tags
There are a few other things about Hit Attributes you may need for particular scenarios, but these are Big Three that every user needs to understand. When you are looking at reports – you’re looking at how often Hit Attributes and their combinations and permutations occur
Event (Basic) – The Hit Attribute “happened”. That is to say, a user submitted or received an interaction with the Web Site that matched the pattern in the Hit Attribute. Now hold this thought – we’ll come back to it in a minute. (of course, Events can get much more complicated, as we’ll see later)
Dimension – Remember how Hit Attribute could have a Start and End tag, and the “value” of the HA was the stuff between the tags (as possibly refined by the RegEx)? You need a way to store these values, and count them. Dimensions define a virtual storage location. Let’s say that in a hypothetical Hit Attribute, there are three different values. And over the course of an hour, the Hit Attribute has appeared 100 times. The Dimension lets us record how often each value appeared. The sum of these counts will equal to 100. Each value of the “dimension” may appear for 0 to 100 times. For example, your site includes a PointOfSale (POS) pattern in response, and there are three values for the POS (us,ca,mx). You could create a POS dimension, its’ values are us, or ca, or mx, and if you record the dimension for a particular event, you would know how often the event happens for each of the three POS values.
We call dimensions “virtual storage” because they define what is to be recorded. The actual storage and reporting is done in Report Groups.
Report Group – Report Groups are created from Dimensions and attached to Events. They provide the actual storage for counting the values of a Hit Attribute. Any Report Group can have 1, 2, 3, or 4 Dimensions. Any Event can have an unlimited number of Report Groups with a single Dimension, but may only have up to four Report Groups with multiple dimensions. This restriction has a big impact on analysis, as we will see later.
Why have both Report Groups and Dimensions? Lets take for example an event that records visits to our product search page, and another event that records visits to the purchase confirmation page. Both pages have the POS pattern. The number of users (and POS distribution) that visit search is different form the number of users (and POS distribution) that visit the purchase confirmation page. The Dimension tells Tealeaf what to record for each event, and attaching the same Report Group to both events provides the two distinct locations for recording the POS distributions.
Session Attribute – A Session Attribute is a named storage location associated with a session. Session attributes can be populated by events or hit attributes. When populated, the previous value of the SA is overwritten. Thus, the SA is said to store “only the last value”. The Session Attributes can be displayed as columns on Search Result pages.

Basic Naming Convention

The tealeaf system has a number of places where event names are displayed in only about 20 characters, and cannot be widened. So it’s important to try and get your events identified ‘early” in their names
The pattern I recommend looks like this:
FunctionalArea:DescriptiveName(R/V):FrequencyOfOccurrence
FunctionalArea – This refers to the “functional area of your site”. I use the letter G to represent events that can happen anywhere (Global). Beyond that, its completely depended on what your site does, and the list will evolve over time as the site evolves. It is a good idea to create a separate document that lists the functional areas, and some identifying characteristics (either a portion of the URL-path, or a specific tag in the response) of each. If you are fortunate, the development team will incorporate information in each Response that specifically identifies the functional area (and even sub-area) to which that response belongs.
In the interest of keeping the event names short and descriptive, a two-letter or three-letter abbreviation of the functional area should be used, in place of a long name. Keep the list of abbreviations in the same document as your functional areas document.
I also recommend using :Err in the FunctionalArea portion to identify events that record error conditions. so G:Err would be errors that could occur anywhere on the site.
DescriptiveName(R/V) – Everybody has different ideas on what should be placed here. I will start by listing some “Do’s and Don’ts”, based on experiences with evolving sites, then go on to what I recommend.
Don’t use the phrase “step1”, or “step2” in an event name. As soon as the developers release a new feature that inserts itself between two existing steps, or they condense two existing steps into a single page view – then all your “step” number have to get changed. Instead of putting “step” in the event names, use the Tealeaf Reports to organize the events into process steps. The exception is creating “pyramid” session events for processes, which we will discuss later.
Do take great pains to create separate events for the Request for a page and the Delivery of the page. For example, if your site has a page /myaccount, one event DescriptiveName could be “MyAcountR” to measure how often users request that URL, and another event could be “MyAcountV” that measures how often the page is delivered (using a tag or pattern in the actual response). Because you have separated the request from the delivery, you can measure the percentage of how often the page is successfully delivered, and you can perform searches to find sessions where a page is requested but NOT delivered. I like to use the letters R and V at the end of each DescriptiveName to indicate this separation. If a DescriptiveName does not end in R or V, then it is usually a combination of other events, or an event that detects a redirect condition.
FrequencyOfOccurrence – the last portion of the name should show a user, at a glance, if the event will fire on every occurrence of the pattern, or if it fires only once in a session. It’s much better to have this in the event name, than having to go look up the event definition to determine it. I recommend the following abbreviations:

:E – EVERY. The event triggers and is recorded on every occurrence.
:S – Session. The event is evaluated at the END of the session, and the timestamp of the event is recorded as occurring at the end of the session.
:O – Once. The event is evaluated on every hit. the event is recorded the FIRST time it occurs. The timestamp of the event is the time of the hit on which it first appears.

The difference between :S and :O is important! if you are measuring “process steps”, and you plan to create conversion metrics from these events, then use :S. All of the events then happen in the same hour. If you want to be able to find something “the first time it happens in a session”, then use :O. In general, use :S for anything you want to record just once, and only use :O when you are searching for just specifically the very first occurrence.

 

Naming Convention for Error Events

In general, I like to use :Err after the FunctionalArea to indicate an error. it is easy to search for :Err when you are searching for specific error events in places where events are selected.
There is another convention I recommend for error events. Errors can be caused by the user making an entry that is “disallowed”, or an error can be caused by the web application faulting. For error events that are trigged by user disallowed activities, I suggest (USR) as part of the event Name, and (SYS) if the event is caused by a failure of the web application.

Naming Convention for Hit Attributes

In general, naming conventions for hit attributes can be much less restrictive then the conventions for event names. Tealeaf users see Hit Attributes only when they are looking at event definitions. With this in mind, you should consider naming the HAs so they make sense when refereed to as event conditions or event values to record. To continue with our example from above, the POS hit attribute could be named POSExtractedFromResponse or POSFromURLVirtualPath. These names would accurately reflect where the hit attribute is drawing its data from. Sometimes, I will use the StartTag, the elide symbol (…) . and the EndTag for the HA name, e.g., class=”FieldError>…</span. While that name may seem a bit weird, it very accurately describes what the HA extracts from the hit.
Don’t prefix your HA with HA, or use the word Hit Attribute in the name. Your wouldn’t call you kid “Child:John” would you? The fact that it is a Hit Attribute name can be inferred from how it is used.
Using the words Request or Response (REQ or RSP) in the HA is a matter of preference. I usually like to have (REQ) or (RSP) at the end of a HA name just to remind myself where it came from, so I use the right one in an event. For example “nameOnCard(REQ)” vs. “nameOnCard(RSP)” would remind me that on the first case, the HA extracts the information from what the user submitted, while in the second case the HA extracts the names from a pattern found somewhere in the response, perhaps a div or span with that name.

Naming Convention for Dimensions

Keep the Dimension names short. You will see later I recommend that Report Group names be constructed from the Dimension names that make up the Report Group. So having long Dimension names leads to unwieldy Report Group names.
Dimension names are shown to users in the Report Builder. They will be dragging these to the X-axis and to the Segment selector to slice event counts. keep them short, and as descriptive as possible, so users’ know what they are seeing.
There are different ways to populate a Dimension, and it makes sense to include something in the Dimension name to indicate to the user what was used to populate the dimension.

• Dimension is populated by a Hit Attribute. When the event fires, the appropriate HA must be present on the hit. If the HA is not present, the value assigned to the dimension is [null]. I like to use :H as the last characters of the Dimension name.
• Dimension is populated by the value of an event, either “first” or “lastest”. “first” is used occasionally, but “latest” is one of the most important ways to populate a dimension. The system “looks back” to the last time an event fired, and records the value of the “lastest” of that event. These are great for  recording the “state” of the end-user’s session. I like to use :E-F and :E-L as the last characters of the Dimension name when they are populated by events 
• Dimension is populated by a Session Attribute. I’ve never found a need to populate a dimesnion from a session attribute. Instead, I populate the dimension from whatever event is populating the SA.

 

Naming Convention for Report Groups

If you construct the Report Group name from the Dimensions used by that Report Group, it makes it very easy to select the appropriate Report Group(s) to assign to an Event.If you wish to report a single Dimension Report Group, I suggest simply copying the Dimension Name and creating the Report Group with exactly the same name. Using Tealeaf’s built-in Report Group creator appends DG_ in front of the Dimension name. I find this prefix to be of little value, so I prefer to simply copy the Dimension name. Back to a previous rhetorical question: you wouldn’t name your dog as DOG_Spot, would you? That a Report Group can be inferred from context means there is no needs to add the DG_ prefix.
Here are some examples: URL:HI/App:HI/Host:HI/Server:HI; URL:HI/POS:E-L; URL:HI/FirstErrorCodeOnPage:E-L/FirstErrorMessageOnPage:E-L
Because the Report Group Name is only used by administrators (Users see only the Dimension names), there is no reason why the RG name can’t be as long as you want, to be descriptive.

Naming Convention for Session Attributes

Session Attributes are good for holding “the last value seen in a session”. They can be displayed on the search results pages, as well as be searched for. The classic example the Tealeaf supplies is “login”. This holds the last value the user entered for a login (assuming the event is properly created).
There are no specific conventions for Session Attributes. With the V8 ability to create events that look back at the value of previous events, the old V7 use of session attributes to hold state is no longer necessary. Events can now detect a change in a field without having to use session attributes as a storage location. They have become simply repositories for information you wish to display on a search results.

Naming Convention for Advanced Mode events

There is a subtle point for the event popups (however over an event and a popup (tooltip) with information about the event appears) that makes it important to note somewhere that an event uses Advanced mode. When you first create an event, you must assign it some condition on which it fires. If you later turn on advanced mode and modify the event, the original condition remains in the event popup! Say you created the event as firing on condition A. If you edit the event in advanced mode, and change it to fire on condition B, the tooltip for that event does not change – it still shows it as being dependent on condition A. Same for the event value. For these reasons, anytime you create an Advanced mode event, I suggest that the event description start with [ADV]. I recommend the description as the place to record this, because it only needs to be present in the same place the incorrect tooltip appears. The event descriptions also appear in those tooltips. Having [ADV] in the description tells the experienced tealeaf user that the conditions or values shown on the tooltip are not necessarily the conditions or values used by the event.

That’s it for naming conventions. I hope you are able to put these to good use in your organization!

Multiple 407 Status Codes when reviewing a Client-Side Capture

 

When looking at a Client-Side capture (either with Fidddler2 or with RealiTea Viewer), at some sites we see the client is making multiple requests for a page, or resource. The sequence is

1. Request for /resource.do
2. Response Status Code 407
3. Request for /resource.do
4. Response Status Code 407
5. Request for /resource.do
6. Response code 200 (and the page is returned)

 

This pattern indicates that there is a proxy between the client and the web server, and the proxy is configured to use a Challenge/Response Authentication scheme. Here are some additional details for that same sequence above:

1. Request for /resource.do
• The browser sends no authentication headers in this request
2. Response Status Code 407
• The proxy responds with a “Proxy Authentication Required” Status code and also tell the web server what kind of authentication is requried, eitehr basic or Challenge/Response
3. Request for /resource.do
• The browser sends the request again, this time with a HTTP header that tells the proxy it wants to use Challenge/Response
4. Response Status Code 407
• The proxy responds with the 407 status code, and additional it includes the Challenge information
5. Request for /resource.do
• The browser sends the request again, this time with a HTTP header that includes the Response to the Challenge
• The proxy lets this request through to the web server
6. Response code 200 (and the page is returned
• The server returns the response to the proxy, which passes it back to the browser.

 

This article by MSDN has further details and  nice explanation http://msdn.microsoft.com/en-us/library/windows/desktop/aa383144(v=vs.85).aspx

Windows Server Setup for Tealeaf Servers

If you are setting up brand new Windows Server 2008 R2 systems for Tealeaf servers, here are the settings and programs you will want to configure. These assume the servers will be in a Workgroup, with no ACtive Directory. Remember – you cannot use a Domain Controller as a tealeaf server, the tealeaf software will not run on a Domain Controller.

Common to All Servers

• Configure the SAS drive that comes with the server so the available drive space is dedicated to C: Drive.
• Install Windows Server 2008 R2 Enterprise OS to the C: Drive.
• Provide information for Workgroup, Server Name, Local Administrator Name when asked
• For the local administrator, use TBD/TBD
• Place all servers in a workgroup TBD (one suggestion would be “Tealeafarm”)
• Provide information for IP Address, Default Gateway, when asked
• Assign the server name and IP address per the corporate standard and communication spreadsheet
• Default Gateway is TBD
• After installation completes, run Windows Updates, then reboot
• Disable the UAC for local administrators
• Disable User Access Control
• Start –> Settings –> Control Panel –> User Accounts –> Turn User Account Control  off
• Run this command (or edit registry by hand) to add a special registry key to make administrators true administrator: C:\Windows\System32\cmd.exe /k %windir%\System32\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /
• Set the following Windows Features (From Control panel add/remove programs->Windows Features)
• Enable Remote Desktop
• Start –> Settings –> Control Panel –> Programs—>Turn Windows Features On –> TBD
• Download, unblock, and install the .Net 3.5 Libraries
• TBD
• Configure a NTP Source
• Windows Time Service and Resulting Internet Communication in Windows Server 2008
• A list of the Simple Network Time Protocol (SNTP) time servers that are available on the Internet
• How to enable and use a local Network Time Protocol (NTP) server
• Configure Windows PowerShell as follows
• Set the Execution policy to unrestricted and enable PS-Remoting by opening a PowerShell Command Prompt and entering the following commands:

Set-ExecutionPolicy  Unrestricted
Enable-PSRemoting -Force

 

• Install the following additional SW for productivity. Install all to their default locations




• GreenShot (Screen Capture) Greenshot V 8.0.0627 Download


• Ditto (Clipboard manager) Ditto V3.17.0.17 Download


• Notepad++ (Enhanced Editor) Notepad++ v5.9.5 Download

HBR Servers (These will communicate with the PCA servers)

• Configure the SAN drive (one LUN, one drive, one TB) as drive D:


• Install the following additional SW for productivity. Install all to their default locations


• Windows Secure File Copy (WinSCP)  WinSCP  V4.3.5 Download


• Putty (SSH for Windows) Putty (Latest Version) Download

Canister Servers

• Configure the SAN drives (two LUNS, two drives, 1 TB each) as drives D: and E:

Image Server

• Configure the SAN drive (one LUN, one drive, one TB) as drive D:

Portal/Reporting Servers

• Configure the SAN drives (two LUNS, two drives, 1 TB each) as drives D: and E:


• Enable and Configure IIS




• 




• Install and Configure SQL Server 2008 R2 Enterprise Version




• TBD