V8 Process Event Guidelines

Every e-commerce web site has one or more “processes” – sequential steps the user has to go through, to accomplish a goal. The most important process for most e-businesses is the “checkout” process (aka the “purchase” process). In this post, we will create discuss how to create events that track processes, and discuss the characteristics you should build into these events.

At the fundamental level, a client sends a request, and the server sends back a response (view). You should create two process events for each step, one to recognize and count the request, and another that recognizes the response.  When tealeaf recognizes a request event, you know the user actively asked for the next step of the process, and when tealeaf recognizes the response, you know the system presented the information to the user. In my post V8 Event Naming Conventions I discuss how to create event names that reflect this difference. The request event can be based on the URL of the page, and the view event  is best based on a unique pattern present in the response when the page is successfully rendered.

However, before we go much further, there are major differences between Apache/IIS/ColdFusion server technology when it comes to how the technology uses the URL, and this difference makes a huge difference in how the Request for a step is recognized. <Insert moew on rthis>

Now that we are all using the same event names to mean the same logical view of the steps regardless of our site’s technology, onward!

Conceptually, a simple checkout process is “Cart View”, “Billing Info”, “Final review” and Confirmation. This produces and eight step checkout processes

Process Step Events for Every Step

The eight steps that measure every step occurrence are P:CartViewR:E, P:CartViewV:E, P:BillingInfoR:E, P:BillingInfoV:E, P:FinalReviewR:E, P:FinalReviewV:E, P:ConfirmationR:E, and P:ConfirmationV:E. When we create the events, base the R events on the URL of the page, and V events on the page title or some other string in the response that positively and uniquely identifies the page. Set the event type as ‘count”, evaluate on every hit, and do not add any report group yet.

These events tell us how many times each process step occurs, even if they occur multiple times in a session.

Session Events for Each Step of the Process

The eight Session steps that measure “the session had one or more occurrences of a process step” are named P:CartViewR:S, P:CartViewV:S, P:BillingInfoR:S, P:BillingInfoV:S, P:FinalReviewR:S, P:FinalReviewV:S, P:ConfirmationR:S, and P:ConfirmationV:S. Create each of these based on the presence of the corresponding :E event being in the session. Set the event type as ‘count”, evaluate at the end of the session, and do not add any report group yet.

These events tell us how many visitors got to each step of the checkout process during their sessions.

Pyramiding the Session Events

Now string the process steps together by creating the following events. It’s easy to see why they are called pyramiding events. Event type is count, evaluate at the end of the session, and no report group. These tell us how many visitors saw each step, and did not miss a step. They are the most accurate source of information for conversion ratios, and will be used for error detection as well, as will be shown later.

P:Step1R:S - P:CartViewR:S

P:Step1R&1V:S - P:CartViewR:S AND P:CartViewV:S

P:Step1R&1V&2R:S - P:CartViewR:S AND P:CartViewV:S AND P:BillingInfoR:S

P:Step1R&1V&2R&2V:S -

P:Step1R&1V&2R&2V&3R:S -

P:Step1R&1V&2R&2V&3R&3V:S -

P:Step1R&1V&2R&2V&3R&3V:S -

P:Step1R&1V&2R&2V&3R&3V&4R:S -

P:Step1R&1V&2R&2V&3R&3V&4R&4V:S - P:CartViewR:S AND P:CartViewV:S AND P:BillingInfoR:S AND P:BillingInfoV:S AND P:FinalReviewR:S AND P:FinalReviewV:S AND P:ConfirmationR:S AND  P:ConfirmationV:S

Simple Count and Conversion Ratio Reports:

It is straightforward to create reports that show the counts for each event, and the ratio between any two of them. You can generate step-step ratios, or ratios from first step to any subsequent step, including confirmation.

In the chart examples throughout this post, the site had a seven-step process, so the pictures do not “quite” match up to the events described in the post. I wanted to use a very short hypothetical process for the text, to keep the post focused on core concepts. I trust you will not have any problem making the correlation and bridging the discrepancies between the charts and the text.

Pyramid Event Counts

Pyramid Event Ratios

 

Dimensions and Report Groups

How can we “slice” these numbers to get focused information? By using Dimensions and Report Groups! Lets examine the possibilities.

Things that are unchanged throughout the session (on the site I’m using for this post) include the AKAMI Country Code and the Session’s First Hit Referrer Domain

Things that may change at any time in the session include the Point Of Sale (POS) which the user can control (it changes the language) and is found in each response, the Currency of the purchase (which is based on the country in which the credit card is issued), and is always present in the CartView response, and the AmountList, which is a grouping, or “bucketing” of the cart’s sale amount. Another item that changes (but usually only once) is the customer’s loyalty program level if they sign-in.

Now, if we want to know the event counts for each country as reported by Akamai, we add a report group to each event that consists of a dimension that is populated by an event that fires on the first hit of the session and whose value is the value of the Akamai Country Code in a http header. Then in the report builder, we can drag the FirstSeenAkamaiCountryCode:E-F dimension to the chart of event counts, and filter on the top five. This groups the events by the country code that was seen on the first request of the session, and tells us which country has the highest number of visits to each step. Do this on the chart of conversion ratios, and it tells us which country has the highest (or lowest) conversion ratios

Now, if we want to know the event counts for referring domain, we add a report group to each event that consists of a dimension that is populated by an event that fires on the first hit of the session and whose value is the value of the domain portion of the HTTP_REFERRER. Then in the report builder, we can drag the DomainSessionReferrer:E-F dimension to the chart of event counts, and filter on the top five. This groups the events by the domain of the referrer that was seen on the first request of the session, and tells us which referrer has the highest number of visits to each step. Do this on the chart of conversion ratios, and it tells us which referrer has the highest (or lowest) conversion ratios.

If we want to know the event counts by POS, we add a report group to each event that consists of a dimension that is populated by an event that extracts (from every hit) the POS from the response, and in the report builder we can drag the POS:E-L dimension to the chart, and filter on the top five. This groups the step events by the POS that was last seen in the session. It is important to note that the POS can change after the last step of the process. The pyramid events are session events, and are evaluated at the end of the session. The value of POS stored in the report group for the pyramid events will be whatever POS was last seen in the last hit of the session.

Two-Dimensional Report Group

When the site is multi-currency, you cannot add the amounts of orders together. Adding Yen to Yuan to Reals to Peso to dollars makes no sense. It is important to consider both the amount and the currency together as a two-part object. We can accomplish this with a report group having two dimensions, Currency:E-L and AmountList:E-L. Populating these dimensions can be done with an advanced mode event. The details are beyond the scope of this already-too-long post, but can be found here. <insert>

Now in the report builder we can drag the Currency:E-L and the AmountList:E-L dimensions to the chart. We lose the graphical representation, but the tabular data shows which currency and amount buckets are the best performing. (I have no idea why there are [Null] entries – looks like I’ve got some troubleshooting to do :-)

Multi-dimension Report Groups

For fine-grained analysis, create report groups of multiple dimensions (up to four dimensions). For example, a report group consisting of the dimensions POS:E-L/LoyaltyStatus:E-L/DomainSessionReferrer:E-F lets us investigate the relationships between these three constraints on the purchase process events.

<insert>

Other dimensions you may want to consider adding are TrafficTypeE-F,BrowserType:-F,BrowserVersion:E-F,SessionDurationList:H

Make sure you look at the TrafficType:E-F dimensions or via searching, to see if there are robots getting to the first part of your process

What is “abandonment”

Abandonment is simply any session that has the first step of the checkout process,and not the last step. Use these two conditions to create a simple session event for P:Abandonment:S Since this means our customer left empty-handed, companies want to spend a fair amount of resources to investigate why this happens.What companies need are a way to quantify how much revenue is lost when an abandonment occurs. For this example, we will focus on the amount (and currency) that was in the last P:CartViewV:E of the session.

Extracting “loss” on an abandonment

Scraping off the amount and the currency from a response can be tough. The details are here <insert xref>.To summarize you need two advanced mode events (BB) with three regular expressions (RE). Both events use one same RE to recognize the entire class element that encloses the the currency and amount substrings. Each BB event uses one of the other two REs, to split the substrings out of the class element string. If you are lucky, the developers have repeated that pattern everywhere. If not, you may be looking at multiple Regular Expressions in these events.

Two dimensions are populated by these BB events StoreLastSeenRevenueCurrency:BB and StoreLastSeenRevenueAmount:BB, called RevenueCurrencyList:E-L and RevenueAmountList:E-L. The dimensions are populated with the “Last” value of the BB event, which itself fires on each occurrence of the event P:CartViewV:E.

The two dimensions can be combined into a single report group and attached to any checkout process event, but wait…. lets not waste a multi-dimension report group. The final report group we attach to abandon will have more dimensions…

The amount and currency that is being abandoned…

Extracting Error Messages

See <insertt xref here> for details. To populate the two dimensions x and Y, It takes two BB events with multiple Regular Expressions. These BB events look at a page for any class, span , or div pattern that contains the error class element, extract either the language-specific error message or the language independent error code, repeats down the whole page view, and aggregates these one or more substrings into one large aggregate string.

You can attach these dimensions to a step event, but they will always have the last seen event value – even if the last seen error was two hits back. Instead, we create an event G:AnyVisibleFieldError:E, and attach the two dimensions to this event along with the URL (Normalized) dimension. This provides an event whose dimensions can tell us “what is the most common error” for any specific URL or groups of URLs. That’s pretty potent! If you attached just these to the P:Abandoned:S event, you would know what error messages were seen last most often when visitors abandoned. Again, lets not waste a multi-dimension report group..

Top Error messages…

To find out which error message was last seen most often and sort by revenue lost, create a report group with all four dimensionss we’ve discussed

a/b/c/d

Now chart the P:Abandoned:E event in the report builder to see the culmination of our efforts which error messages are seen for the largest amount of abandoned shopping carts, grouped by currency and amount list and errormessage (and errorcode)…

Throw in the revenue amount list to group these

 

The P:Abandoned Event

• Evaluate at End of Session
• Fires if teh session has a ProcessStart and NOT a ProcessEnd event
• Numeric Type of value
• Store the value amount shown at ProcessStart
• Report group contains RevenueCurency:E-L; RevenueAmount:E-L and AggregateVisibleFieldErrorMessage:E-L

I hope this has helped you understand process events, and given you some ideas for expanding your use of tehm.

V8 Event Naming Conventions

With the introduction of the new event models in Version 8 of Tealeaf, new capabilities  and terminology require another look at how events are named. Event naming is strictly “by convention” – you can name an event anything you want, but after you have 500+ events, finding one again a few months later is much easier if the event names all follow some conventions.
Good conventions means that anybody who looks at an event name can get a very good idea of what it does. Good conventions reduce maintenance cost by reducing the number of duplicated events created by different event administrators, and good conventions make it easy to pass along the institutional knowledge when new event administrators are brought on board.  

Terminology and Definitions

All licensed users have access to the online documentation appropriate to their version of tealeaf. One very useful piece of this documentation is the Glossary of Terms. Everyone in your organization who uses tealeaf or consumes the reports generated by  tealeaf should review this document so that everyone shares the same terminology. To access the Glossary, Start the Portal, enter “Glossary” in the Search Online Help and search. While the results page is empty (as of V8.45), enter Glossary again in “Site Search All Spaces”, search, and you will be shown links to the tealeaf glossary.
The important terms, from an event perspective, are:
Hit Attribute: The pattern you are looking for! This is the foundation of the whole eventing system. Three important things:
1) Text String: - a fixed constant pattern of characters, in either the Request or the Response
2) Start Tag/End Tag – a constant pattern of characters that make up the start of the pattern (Start Tag), and another constant pattern of characters that make up the end of the pattern (End Tag). Everything between the Start Tag and End Tag make up the '”value” of that particular Hit Attribute
3) RegEx Pattern – a way to better refine the data between the Start/End tags. Using RegEx (or RegExpr) patterns and match groups, you can pull out just specific portions of the character strings between the Start and End tags, and give the Hit Attribute the value of just some substring(s) of the full data between the Start/End tags
There are a few other things about Hit Attributes you may need for particular scenarios, but these are Big Three that every user needs to understand. When you are looking at reports – you’re looking at how often Hit Attributes and their combinations and permutations occur
Event (Basic) – The Hit Attribute “happened”. That is to say, a user submitted or received an interaction with the Web Site that matched the pattern in the Hit Attribute. Now hold this thought – we’ll come back to it in a minute. (of course, Events can get much more complicated, as we’ll see later)
Dimension – Remember how Hit Attribute could have a Start and End tag, and the “value” of the HA was the stuff between the tags (as possibly refined by the RegEx)? You need a way to store these values, and count them. Dimensions define a virtual storage location. Let’s say that in a hypothetical Hit Attribute, there are three different values. And over the course of an hour, the Hit Attribute has appeared 100 times. The Dimension lets us record how often each value appeared. The sum of these counts will equal to 100. Each value of the “dimension” may appear for 0 to 100 times. For example, your site includes a PointOfSale (POS) pattern in response, and there are three values for the POS (us,ca,mx). You could create a POS dimension, its’ values are us, or ca, or mx, and if you record the dimension for a particular event, you would know how often the event happens for each of the three POS values.
We call dimensions “virtual storage” because they define what is to be recorded. The actual storage and reporting is done in Report Groups.
Report Group – Report Groups are created from Dimensions and attached to Events. They provide the actual storage for counting the values of a Hit Attribute. Any Report Group can have 1, 2, 3, or 4 Dimensions. Any Event can have an unlimited number of Report Groups with a single Dimension, but may only have up to four Report Groups with multiple dimensions. This restriction has a big impact on analysis, as we will see later.
Why have both Report Groups and Dimensions? Lets take for example an event that records visits to our product search page, and another event that records visits to the purchase confirmation page. Both pages have the POS pattern. The number of users (and POS distribution) that visit search is different form the number of users (and POS distribution) that visit the purchase confirmation page. The Dimension tells Tealeaf what to record for each event, and attaching the same Report Group to both events provides the two distinct locations for recording the POS distributions.
Session Attribute – A Session Attribute is a named storage location associated with a session. Session attributes can be populated by events or hit attributes. When populated, the previous value of the SA is overwritten. Thus, the SA is said to store “only the last value”. The Session Attributes can be displayed as columns on Search Result pages.

Basic Naming Convention

The tealeaf system has a number of places where event names are displayed in only about 20 characters, and cannot be widened. So it’s important to try and get your events identified ‘early” in their names
The pattern I recommend looks like this:
FunctionalArea:DescriptiveName(R/V):FrequencyOfOccurrence
FunctionalArea – This refers to the “functional area of your site”. I use the letter G to represent events that can happen anywhere (Global). Beyond that, its completely depended on what your site does, and the list will evolve over time as the site evolves. It is a good idea to create a separate document that lists the functional areas, and some identifying characteristics (either a portion of the URL-path, or a specific tag in the response) of each. If you are fortunate, the development team will incorporate information in each Response that specifically identifies the functional area (and even sub-area) to which that response belongs.
In the interest of keeping the event names short and descriptive, a two-letter or three-letter abbreviation of the functional area should be used, in place of a long name. Keep the list of abbreviations in the same document as your functional areas document.
I also recommend using :Err in the FunctionalArea portion to identify events that record error conditions. so G:Err would be errors that could occur anywhere on the site.
DescriptiveName(R/V) – Everybody has different ideas on what should be placed here. I will start by listing some “Do’s and Don’ts”, based on experiences with evolving sites, then go on to what I recommend.
Don’t use the phrase “step1”, or “step2” in an event name. As soon as the developers release a new feature that inserts itself between two existing steps, or they condense two existing steps into a single page view – then all your “step” number have to get changed. Instead of putting “step” in the event names, use the Tealeaf Reports to organize the events into process steps. The exception is creating “pyramid” session events for processes, which we will discuss later.
Do take great pains to create separate events for the Request for a page and the Delivery of the page. For example, if your site has a page /myaccount, one event DescriptiveName could be “MyAcountR” to measure how often users request that URL, and another event could be “MyAcountV” that measures how often the page is delivered (using a tag or pattern in the actual response). Because you have separated the request from the delivery, you can measure the percentage of how often the page is successfully delivered, and you can perform searches to find sessions where a page is requested but NOT delivered. I like to use the letters R and V at the end of each DescriptiveName to indicate this separation. If a DescriptiveName does not end in R or V, then it is usually a combination of other events, or an event that detects a redirect condition.
FrequencyOfOccurrence – the last portion of the name should show a user, at a glance, if the event will fire on every occurrence of the pattern, or if it fires only once in a session. It’s much better to have this in the event name, than having to go look up the event definition to determine it. I recommend the following abbreviations:

:E – EVERY. The event triggers and is recorded on every occurrence.
:S – Session. The event is evaluated at the END of the session, and the timestamp of the event is recorded as occurring at the end of the session.
:O – Once. The event is evaluated on every hit. the event is recorded the FIRST time it occurs. The timestamp of the event is the time of the hit on which it first appears.

The difference between :S and :O is important! if you are measuring “process steps”, and you plan to create conversion metrics from these events, then use :S. All of the events then happen in the same hour. If you want to be able to find something “the first time it happens in a session”, then use :O. In general, use :S for anything you want to record just once, and only use :O when you are searching for just specifically the very first occurrence.

 

Naming Convention for Error Events

In general, I like to use :Err after the FunctionalArea to indicate an error. it is easy to search for :Err when you are searching for specific error events in places where events are selected.
There is another convention I recommend for error events. Errors can be caused by the user making an entry that is “disallowed”, or an error can be caused by the web application faulting. For error events that are trigged by user disallowed activities, I suggest (USR) as part of the event Name, and (SYS) if the event is caused by a failure of the web application.

Naming Convention for Hit Attributes

In general, naming conventions for hit attributes can be much less restrictive then the conventions for event names. Tealeaf users see Hit Attributes only when they are looking at event definitions. With this in mind, you should consider naming the HAs so they make sense when refereed to as event conditions or event values to record. To continue with our example from above, the POS hit attribute could be named POSExtractedFromResponse or POSFromURLVirtualPath. These names would accurately reflect where the hit attribute is drawing its data from. Sometimes, I will use the StartTag, the elide symbol (…) . and the EndTag for the HA name, e.g., class=”FieldError>…</span. While that name may seem a bit weird, it very accurately describes what the HA extracts from the hit.
Don’t prefix your HA with HA, or use the word Hit Attribute in the name. Your wouldn’t call you kid “Child:John” would you? The fact that it is a Hit Attribute name can be inferred from how it is used.
Using the words Request or Response (REQ or RSP) in the HA is a matter of preference. I usually like to have (REQ) or (RSP) at the end of a HA name just to remind myself where it came from, so I use the right one in an event. For example “nameOnCard(REQ)” vs. “nameOnCard(RSP)” would remind me that on the first case, the HA extracts the information from what the user submitted, while in the second case the HA extracts the names from a pattern found somewhere in the response, perhaps a div or span with that name.

Naming Convention for Dimensions

Keep the Dimension names short. You will see later I recommend that Report Group names be constructed from the Dimension names that make up the Report Group. So having long Dimension names leads to unwieldy Report Group names.
Dimension names are shown to users in the Report Builder. They will be dragging these to the X-axis and to the Segment selector to slice event counts. keep them short, and as descriptive as possible, so users’ know what they are seeing.
There are different ways to populate a Dimension, and it makes sense to include something in the Dimension name to indicate to the user what was used to populate the dimension.

• Dimension is populated by a Hit Attribute. When the event fires, the appropriate HA must be present on the hit. If the HA is not present, the value assigned to the dimension is [null]. I like to use :H as the last characters of the Dimension name.
• Dimension is populated by the value of an event, either “first” or “lastest”. “first” is used occasionally, but “latest” is one of the most important ways to populate a dimension. The system “looks back” to the last time an event fired, and records the value of the “lastest” of that event. These are great for  recording the “state” of the end-user’s session. I like to use :E-F and :E-L as the last characters of the Dimension name when they are populated by events 
• Dimension is populated by a Session Attribute. I’ve never found a need to populate a dimesnion from a session attribute. Instead, I populate the dimension from whatever event is populating the SA.

 

Naming Convention for Report Groups

If you construct the Report Group name from the Dimensions used by that Report Group, it makes it very easy to select the appropriate Report Group(s) to assign to an Event.If you wish to report a single Dimension Report Group, I suggest simply copying the Dimension Name and creating the Report Group with exactly the same name. Using Tealeaf’s built-in Report Group creator appends DG_ in front of the Dimension name. I find this prefix to be of little value, so I prefer to simply copy the Dimension name. Back to a previous rhetorical question: you wouldn’t name your dog as DOG_Spot, would you? That a Report Group can be inferred from context means there is no needs to add the DG_ prefix.
Here are some examples: URL:HI/App:HI/Host:HI/Server:HI; URL:HI/POS:E-L; URL:HI/FirstErrorCodeOnPage:E-L/FirstErrorMessageOnPage:E-L
Because the Report Group Name is only used by administrators (Users see only the Dimension names), there is no reason why the RG name can’t be as long as you want, to be descriptive.

Naming Convention for Session Attributes

Session Attributes are good for holding “the last value seen in a session”. They can be displayed on the search results pages, as well as be searched for. The classic example the Tealeaf supplies is “login”. This holds the last value the user entered for a login (assuming the event is properly created).
There are no specific conventions for Session Attributes. With the V8 ability to create events that look back at the value of previous events, the old V7 use of session attributes to hold state is no longer necessary. Events can now detect a change in a field without having to use session attributes as a storage location. They have become simply repositories for information you wish to display on a search results.

Naming Convention for Advanced Mode events

There is a subtle point for the event popups (however over an event and a popup (tooltip) with information about the event appears) that makes it important to note somewhere that an event uses Advanced mode. When you first create an event, you must assign it some condition on which it fires. If you later turn on advanced mode and modify the event, the original condition remains in the event popup! Say you created the event as firing on condition A. If you edit the event in advanced mode, and change it to fire on condition B, the tooltip for that event does not change – it still shows it as being dependent on condition A. Same for the event value. For these reasons, anytime you create an Advanced mode event, I suggest that the event description start with [ADV]. I recommend the description as the place to record this, because it only needs to be present in the same place the incorrect tooltip appears. The event descriptions also appear in those tooltips. Having [ADV] in the description tells the experienced tealeaf user that the conditions or values shown on the tooltip are not necessarily the conditions or values used by the event.

That’s it for naming conventions. I hope you are able to put these to good use in your organization!

Multiple 407 Status Codes when reviewing a Client-Side Capture

 

When looking at a Client-Side capture (either with Fidddler2 or with RealiTea Viewer), at some sites we see the client is making multiple requests for a page, or resource. The sequence is

1. Request for /resource.do
2. Response Status Code 407
3. Request for /resource.do
4. Response Status Code 407
5. Request for /resource.do
6. Response code 200 (and the page is returned)

 

This pattern indicates that there is a proxy between the client and the web server, and the proxy is configured to use a Challenge/Response Authentication scheme. Here are some additional details for that same sequence above:

1. Request for /resource.do
• The browser sends no authentication headers in this request
2. Response Status Code 407
• The proxy responds with a “Proxy Authentication Required” Status code and also tell the web server what kind of authentication is requried, eitehr basic or Challenge/Response
3. Request for /resource.do
• The browser sends the request again, this time with a HTTP header that tells the proxy it wants to use Challenge/Response
4. Response Status Code 407
• The proxy responds with the 407 status code, and additional it includes the Challenge information
5. Request for /resource.do
• The browser sends the request again, this time with a HTTP header that includes the Response to the Challenge
• The proxy lets this request through to the web server
6. Response code 200 (and the page is returned
• The server returns the response to the proxy, which passes it back to the browser.

 

This article by MSDN has further details and  nice explanation http://msdn.microsoft.com/en-us/library/windows/desktop/aa383144(v=vs.85).aspx

Windows Server Setup for Tealeaf Servers

If you are setting up brand new Windows Server 2008 R2 systems for Tealeaf servers, here are the settings and programs you will want to configure. These assume the servers will be in a Workgroup, with no ACtive Directory. Remember – you cannot use a Domain Controller as a tealeaf server, the tealeaf software will not run on a Domain Controller.

Common to All Servers

• Configure the SAS drive that comes with the server so the available drive space is dedicated to C: Drive.
• Install Windows Server 2008 R2 Enterprise OS to the C: Drive.
• Provide information for Workgroup, Server Name, Local Administrator Name when asked
• For the local administrator, use TBD/TBD
• Place all servers in a workgroup TBD (one suggestion would be “Tealeafarm”)
• Provide information for IP Address, Default Gateway, when asked
• Assign the server name and IP address per the corporate standard and communication spreadsheet
• Default Gateway is TBD
• After installation completes, run Windows Updates, then reboot
• Disable the UAC for local administrators
• Disable User Access Control
• Start –> Settings –> Control Panel –> User Accounts –> Turn User Account Control  off
• Run this command (or edit registry by hand) to add a special registry key to make administrators true administrator: C:\Windows\System32\cmd.exe /k %windir%\System32\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /
• Set the following Windows Features (From Control panel add/remove programs->Windows Features)
• Enable Remote Desktop
• Start –> Settings –> Control Panel –> Programs—>Turn Windows Features On –> TBD
• Download, unblock, and install the .Net 3.5 Libraries
• TBD
• Configure a NTP Source
• Windows Time Service and Resulting Internet Communication in Windows Server 2008
• A list of the Simple Network Time Protocol (SNTP) time servers that are available on the Internet
• How to enable and use a local Network Time Protocol (NTP) server
• Configure Windows PowerShell as follows
• Set the Execution policy to unrestricted and enable PS-Remoting by opening a PowerShell Command Prompt and entering the following commands:

Set-ExecutionPolicy  Unrestricted
Enable-PSRemoting -Force

 

• Install the following additional SW for productivity. Install all to their default locations




• GreenShot (Screen Capture) Greenshot V 8.0.0627 Download


• Ditto (Clipboard manager) Ditto V3.17.0.17 Download


• Notepad++ (Enhanced Editor) Notepad++ v5.9.5 Download

HBR Servers (These will communicate with the PCA servers)

• Configure the SAN drive (one LUN, one drive, one TB) as drive D:


• Install the following additional SW for productivity. Install all to their default locations


• Windows Secure File Copy (WinSCP)  WinSCP  V4.3.5 Download


• Putty (SSH for Windows) Putty (Latest Version) Download

Canister Servers

• Configure the SAN drives (two LUNS, two drives, 1 TB each) as drives D: and E:

Image Server

• Configure the SAN drive (one LUN, one drive, one TB) as drive D:

Portal/Reporting Servers

• Configure the SAN drives (two LUNS, two drives, 1 TB each) as drives D: and E:


• Enable and Configure IIS




• 




• Install and Configure SQL Server 2008 R2 Enterprise Version




• TBD

How to update the Robot Detection files and lists

I’ve had a some tealeaf administrators ask me where to find the instructions for updating the Robot Detection and Mobile Devices list, so I decided to post my answers here for anyone else interested.

Start by getting the full documentation set for Tealeaf, if you don’t already have it. It’s a great way to have all the tealeaf doc in a single place.

You can get the full documentation set from community.tealeaf.com. If you don’t have a login setup there yet, I’d suggest you get a login there and explore it – there is a ton of resources, and a pretty active community forum where you can post questions and get answers. Once you have logged in, click the Online Help link. The down the left nav bar, you will find Product Docs. Click the highest release number you see visible, and you will see a screen similar to the following, which is the top-level menu for all the documents for a specific release. I’ve highlighted the link that gets you a zip file of all the documents in PDF form.

 

Within this zip, open the file “cxImpact_Administration_Manual_7.2.pdf”

Find Chapter 10. APPENDIX: MANAGING USER AGENTS, and read this chapter for the theory on how bot and mobile detection works.

Next, from that same zip file, open the document “CX_Installation_And_Configuration_Manual_7.2.pdf”

Find “Chapter 7: Maintaining the CX System, subsection UPDATING USER AGENT FILES” (pp 479 on my copy)

Follow the instructions to update the browscap file and the Event Value lists. The doc is a bit weak, so look at your existing Event Value lists for Browser, BrowserVersion, and Platform (in event editor) until you can recognize the patterns in each file, and then experiment with the UserAgentRevealer.exe settings, so you know what settings produces each browser list. Then export your browser lists. One shortcut the document doesn’t mention; UserAgentRevealer.exe produces CSV files, and the event editor list import feature expects txt files. If you simply rename the .CSV output to .TXT, you can import it to the event list. Event list imports are limited to 1000 items, so you may have to import the txt file once, then wack off the top 1000 items, import it again, repeat, until you get all the items imported. The importer recognizes and ignores duplicates, so this approach is fine.

It is important that you take some time to study each of the browser list (Browser, BrowserVersion, platform) in the event editor lists tab, and take some time to study the various options on the UserAgentRevealer.exe, to be sure you match correctly the CSV output of the UserAgentRevealer.txt to the correct event editor list.

PowerShell function to compare registry keys across multiple computers

 

The Tealeaf registry settings can and should be controlled by the Tealeaf Management Service (TMS). However, I’ve occasionally had some problems with TMS not completing, and I have been left wondering if all the Tealeaf keys, properties, sub-keys, and their properties, are correctly in sync across multiple computers. Until TMS gets a “validate” feature, I’ve developed a PowerShell script that will report on any discrepancies between the Tealeaf keys across multiple computers.

To use this, the Tealeaf computers will need to enable Powershell-Remoting. There are plenty of instructions for this on the Internet, and I plan to record a blog here shortly on how to do that.

Remember to define the $computerNames array. It only makes sense to compare Canisters to Canisters, and HBRs to HBRs, so you will probably need to create two lists of names, and run the Compare-AllRegKey function twice.

Here is the code. Paste it into a PowerShell window, and the Help is included. Enjoy!

<#
.SYNOPSIS 
Compares Registry Key Properties and subkeys across multiple computers
.DESCRIPTION
The function Get-AllRegKey  will recurse down from a given key, returning an array having
    the key's properties, subkeys, and their properties and subkeys.
Provide Get-AllRegKey a list of computernames, and it will remote to those computers
    and return the properties, etc. of the same key on the remote computer
Provide function Compare-AllRegKey with the name of the reference computer, a list of
    two or more computer names, and it will call Get-ALlRegKey to retrieve the key 
    information from all the listed computers, then use Compare-Object to return 
    just the differences.
If you want more control over the Compare-Object step, you should modify the 
    function (suggestions welcome for an efficient/concise way to add Compare-Object 
    parameters to the Compare-AllRegKey function)
  
.PARAMETER ComputerNames
In the Get-AllRegKey function this is a single computer name or an array of computer names
In the Compare-AllRegKey function, this must be an array of at least two computer names
.PARAMETER RegKey
a single registry key/hive from which recursion starts, using the Registry Provider syntax
The value defaults to the current-scoped variable $DefaultRegistryKey
.PARAMETER -ReferenceObject
Applies only to the Compare-AllRegKey function, this parameter identifies the computer
against which the other computers are compared. This string must be one of the computer
names found in the ComputerNames parameter.
.EXAMPLE
C:\PS> Get-AllRegKey
.EXAMPLE
C:\PS> Get-AllRegKey -RegistryKey 'HKLM:\SOFTWARE\Microsoft\PowerShell'
.EXAMPLE
C:\PS> Get-AllRegKey -ComputerNames @('localhost','RemoteCN')
.EXAMPLE
C:\PS> Get-AllRegKey -ComputerNames @('localhost','Computer1','Computer2')
.EXAMPLE
C:\PS> Get-AllRegKey localhost 'HKLM:\SOFTWARE\Microsoft\PowerShell'
.EXAMPLE
C:\PS> Compare-AllRegKey -ComputerNames @('localhost','RemoteCN') 
.EXAMPLE
C:\PS> Compare-AllRegKey -ComputerNames @('CN1','CN2') -ReferenceObject CN2
#>

################################################################################
# Default values for the Registry Key 
$DefaultRegistryKey='HKLM:\SOFTWARE\Wow6432Node\TeaLeaf Technology'

################################################################################
# The scriptblock that does the actual work of creating an object representing a 
#  registry key and all it's properties and subkeys and their properties. No Defaults
#  This is recursive and remoteable
$_getRegKeySB = {Param($RegistryHive)
  # Create a local named function
  function _getRegKey {
    Param($RegistryHive)
    # $data is an array, local to each loop of the recursion
    #    initialize it with the name of the hive/key
    $data=@($RegistryHive)
    # Get the hive/keys properties, excluding the ones added by PS
    $props = Get-ItemProperty -Path $RegistryHive | 
        Select * -Exclude PS*Path,PSChildName,PSDrive,PSProvider
    # if $props is empty, piping it to get-member produces an error message
    # so test it for non-null first
    if ($props) {
      $props = $props | get-member -memberType NoteProperty
      # prepend each property with the full name of the key, and add it to $data
      foreach ($p in $props) {$data+=("$RegistryHive`:"+$p.Definition)}
    }
    # recursivly call the same algorithm for any subkeys of the hive/key
    foreach ($sk in (get-item $RegistryHive).GetSubKeyNames()) {
      # if there are any subkeys, append their data to the current data.
      # Use the full name of the key
      $data += (&_getRegKey (($RegistryHive)+'\'+ $sk))
    }
    # the local named function's output is the array representation of the hive/key
    $data
  }
  # Call the local named function
  &_getRegKey $RegistryHive
}

################################################################################
# Across all computers, get the key and subkeys from the registry
#  returns a hash of array objects, keyed by computer name
function Get-AllRegKey {
  Param (
    # Single computer name or an array of computer names. 
    #  Defaults to the "current scoped variable by the same name"
    $computerNames = $computerNames
    # A valid key
    ,$RegistryKey = $DefaultRegistryKey
  )
  # create the empty hash
  $AllRegKey = @{}
  # iterate over each computer name
  foreach ($cn in $computerNames) {
    switch ($cn) {
      # If the computer name is localhost, or the same name as hostname
      #   use the Call operator to call the scriptblock, and assign the array returned 
      #   to the hash using the current computername as the key
      {$_ -match "localhost|" + (hostname)} {
        $AllRegKey.$cn = &$_getRegKeySB $RegistryKey
        break
      }
      # for all other computer names, execute the command remotely using invoke-command
      default {
        # pass the scriptblock to the remote computers
        #  assign the array returned to the hash using the current computername as the key
        $AllRegKey.$cn = (invoke-command -Scriptblock $_getRegKeySB `
            -ArgumentList $RegistryKey -computername "$cn")
      }
    }
  }
  #return the hash of arrays
  $AllRegKey
}

################################################################################
# Across all computers, get the key and subkeys from the registry
#  returns a hash of array objects, keyed by computer name
function Compare-AllRegKey {
  Param (
   # Must be an array, with 2 or more members; 
   #   defaults to the "current scoped variable by the same name"
    $computerNames = $computerNames
    # A valid key
    ,$RegistryKey = $DefaultRegistryKey
    # The name of the computer to use as the reference 
    ,$ReferenceObject
  )
Begin {
  # If the argument for $ReferenceObject is null, then default 
  #    to the first element of $computerNames
  if (!$ReferenceObject) {$ReferenceObject =$computerNames[0]}
  else {
    # Validate that the $referenceObject is an element of $computerNames
    if (!($computerNames -contains $ReferenceObject)) {
      throw ("{0} is not a member of the list {1}" -f $ReferenceObject, `
         ($computerNames -join ','))}
  }
  # Get the Registry Key data for all computers 
  $AllRegKey = Get-AllRegKey $computerNames $RegistryKey

} 
Process {
  # Iterate over the computernames, excluding the $ReferenceObject
  $diff = @{};
  foreach ($cn in $computerNames | Where {$cn -ne $ReferenceObject}) {
    # compare $ReferenceObject to the remaining objects, accumulate into $diff
    $diff.$cn = Compare-Object -ReferenceObject $AllRegKey.$ReferenceObject `
        $AllRegKey.$cn | Where-Object { `
            ($_.SideIndicator -eq '=>') -or ($_.SideIndicator -eq '<=') }
  }
  # Return the difference hash
  $diff
}
}

# The following lines will compare the default registry key 
#  across the default array of computernames
Compare-AllRegKey 

Technorati Tags: PowerShell,TeaLeaf,TMS,Registry

PowerShell Custom Object Factory

I needed to create custom objects with specific properties for programming against a WPF GUI, because WPF programming is simplest when it binds properties of controls to properties of an object. But I loathed the thought of writing a function to create a custom object, and having to duplicate the parameter name as a Property name in the new-object call. I also wanted to have default values for the object’s Properties.

The following solution is a template – Use it when you need a New-BlahBlahObject function. Simply add the parameters you want to the Param() block, and this function will hand you back a PSObject with the Parameters turned into Properties. As a bonus, the ParameterSetName property allows you to use this as a kind of object factory – depending on which arguments you supply, different Properties will get created, and the value of the ParameterSetName will tell you which properties are present.

You can do a lot more customization before the call to new-object in the last line of the function, by modifying the $p2 string. You can also add functions to the object before you return it, like Add, so you can “add” two of these custom objects together.

[Author’s note: If anybody know how to add a “code” widget to a Blogger Blog, please tell me so in a comment!. The following formatting is terrible… I hope word-wrap doesn’t prevent cut’n’paste from working]

function New-CustomDemoObject {
  [CmdletBinding(DefaultParametersetName='All')]
  Param (
   [parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $Workbook = 'T1.xlsx'
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][switch] $isVisible
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][switch] $isKeepOpen
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][string] $SortDirection = 'Ascending'
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][string[]] $DatabaseConnectionStrings = @('connecstr1','connectstr2')
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $ParameterWithDefaultArray = @('connecstr5','connectstr6')
  ,[parameter(ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)][hashtable] $CommandStringsHash  = @{cmd1=@{cmdstr='a command';vo=1;so=2};cmd2=@{cmdstr='b command';vo=2;so=1}}
  ,[parameter(ParameterSetName='PathToDir',ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $PathToDir = 'C:/Logs'
  ,[parameter(ParameterSetName='PathToZip',ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $PathToZip = 'C:/Logs/Backups'
  ,[parameter(ParameterSetName='PathToZip',ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $PathInZip = './'
  ,[parameter(Position=0, ValueFromRemainingArguments=$true)] $RemainingArgs
  )
  # Snippet that creates an object from the parameters
  # Create a list of the common parameters as a RegExp match string
  $cp = iex 'function gcp{[CmdletBinding()]Param()$p1=@();((gci Function:$($pscmdlet.MyInvocation.MyCommand)).Parameters).Keys|%{$p1+=$_};$p1|%{"^$_`$"}};[string]::join("|",(gcp))'
  # Get a list of this function's parameters, eliminate the commonparameters, make a hash with the parameter name and type
  $p1=@{};((gci Function:$($pscmdlet.MyInvocation.MyCommand)).Parameters).Keys|?{$_ -notmatch $cp}|%{$p1.$_=((gci Function:$($pscmdlet.MyInvocation.MyCommand)).Parameters).$_.ParameterType.Name}
  # Iterate this function's real parameters, turn switches to bools, remove ActionPreference (and you can do whatever else you want to in this loop)
  $p2=$p1.Clone();$p1.Keys|%{$key=$_;switch ($p1.$_) {'SwitchParameter' {$p2.$key='Bool';break}'ActionPreference' {$p2.Remove($key);break}}}
  # Add the parameterset as a property (Tells you which parameters are present), and create a new object using the hash of argument names and values
  # This version of the last line creates strongly-typed Properties, but won't handle Parameters/Properties of type arrary or hashtable
  # new-object psObject -Property (&(iex $('{@{ParameterSet=[string]'+ "'$($PsCmdlet.ParameterSetName)';" +[string]::join(";",($p2.Keys|%{("{0}="-f $_+'['+$p2.$_+']'+'"$'+$_ +'"')}))+'}}')))
  # This version of the last line creates untyped Properties, and does handle hashtables and arrays. see also this blog https://jamesone111.wordpress.com/2011/01/11/why-it-is-better-not-to-use-powershell-parameter-validation/
  new-object psObject -Property (&(iex $('{@{ParameterSet=[string]'+"'$($PsCmdlet.ParameterSetName)';"+[string]::join(";",($p2.Keys|%{("{0}="-f $_+"`$$_")}))+'}}')))
}

$newobj1 = New-CustomDemoObject # If none of the specific ParamterSet arguments are supplied, the default ParamterSetName 'All' causes all parameters to be created on the PsObject as empty Properties
$newobj2 = New-CustomDemoObject -Workbook 'another.xlsx' -PathToDir 'D:\Logs'
$newobj3 = New-CustomDemoObject -isVisible -PathToZip 'D:\Logs\Backups'
#$newobj4 = New-CustomDemoObject -Workbook 'another.xlsx' -PathToZip 'D:\Logs\Backups' -PathToDir 'D:\Logs' # Uncomment this - you will see an error about paramterset cannot be resolved

# Type $newobj1 ( and $newobj2, and  $newobj3) at the comamnd prompt to see the objects that got created