Data Profile Tool

I’ve written in this blog before about how I’m often called on to validate a new data file from a client or to examine a new data extract. My first step is to load data into a copy of Steve Dark’s Data Profiler (Steve's original post is useful - search on quickintelligence data profiler). The data profiler gives me information about exactly what values exist in each of the fields.

I’ve made a few modifications to Steve’s original document:

• Added a table box that adjusts itself to the fieldnames in the file
• Added a small macro that updates the Statistics Box with the fieldname automatically as different fieldnames are selected. A trigger detects the change and automatically configures the Statistics Box
• Added a tab to help search for duplicate values in either a field or combination of fields
• A check in the loadscript to see if the file exists and if it does not exist then it opens a msgbox from the loadscript using a function

You can download an example copy of the file data profiler that I use by clicking HERE

.

I also keep a version of the data profiler that is tailored for loading data from an Oracle table and a version of the profiler tailored for loading data from a qvd file. I think I use the data profiler tools almost every day to examine data in files and tables to help with QlikView document design and other reporting requirements.

These two examples have no data loaded - you will have to do a reload yourself from your own data:

• You can download an example copy of the profiler I use for qvd files by clicking HERE
• You can download an example copy of the profiler I use for Oracle tables by clicking HERE

  ★★★

Update Document Variables Directly From Excel Spreadsheet

A few years ago I published a blog entry that showed how to copy all of the variables and their values from a QlikView document into an Excel spreadsheet (search on QlikView Maven Listing All Variables and Contents). It was very useful for me, I use it all the time especially when I want to search or sort the variables. Here is the companion to that piece - it is an Excel macro that will copy the data from your spreadsheet into the document variables of a QlikView document. I will say that again - it is an Excel VBA macro (not a QlikView macro).

Copy and paste to install the macro into your Excel spreadsheet and build a button to execute it. You will find it convenient to insert it into the same Excel file that you use for the variable listing I mentioned above. The macro will use a browse-for-file feature so you can select the QlikView document file you want to update. Then, the macro will go down the rows of your active worksheet taking variable names from column A and variable contents from column B and updating the document until it hits an empty cell in column A. It will then tell you how many variables it has processed and ask if you want to save the document. It starts from row 4 of your  worksheet to make it compatible with the earlier spreadsheet macro but you can change the row that it starts with - look for the comment in the middle of the macro.

As always, make a backup copy of your files before running a macro like this. Here's the macro code:


Sub Update_QV_Variables()
'------------------------
'This Excel macro will ask you to select a QlikView document file and
'then it will insert/updateput the variable names and contents from
'your current worksheet into the QlikView document.
'Variable name must be in column A and variable contents in column B.
'The process goes down the rows until it hits an empty cell in column A.
'--QlikView Maven, April 2013
Dim oQV, oDoc, oVars, oTempVar, oThisVar, qv_fn, i, rowlmt, start_row, varctr, x

'Get the QlikView document pathname
qv_fn = Application.GetOpenFilename("QlikView Report File (.),*.*", , "Select a QlikView .qvw Report File", False)
If qv_fn = 0 Then
  Exit Sub
  End If

'Open the QlikView document file
On Error Resume Next
Set oQV = CreateObject("QlikTech.QlikView")
If Err.Number > 0 Then
  x = MsgBox("Sorry, I can't find QlikView", vbOKOnly, "Update_QV_Variables")
  Exit Sub
  End If
Set oDoc = oQV.OpenDoc(qv_fn)
If Err.Number > 0 Then
  oQV.Quit
  x = MsgBox("Sorry, I can't find the QlikView document", vbOKOnly, "Update_QV_Variables")
  oQV.Quit
  Exit Sub
  End If
Set oVars = oDoc.GetVariableDescriptions
If Err.Number > 0 Then
  oQV.Quit
  x = MsgBox("Sorry, I can't find the QlikView document", vbOKOnly, "Update_QV_Variables")
  Exit Sub
  End If
On Error GoTo 0

start_row = 4 'this is the first spreadsheet row with variable data - change as needed
rowlmt = ActiveSheet.UsedRange.Rows.Count
varctr = 0

'Loop through the rows
For i = start_row To rowlmt
  varname = ActiveSheet.Cells(i, 1).Value
  If Len(Trim(varname)) = 0 Then
    Exit For
    End If
  varctr = varctr + 1
  varcontent = ActiveSheet.Cells(i, 2).Value
  Set oThisVar = oDoc.Variables(varname)
    If oThisVar Is Nothing Then
    'must need to create variable
    oDoc.CreateVariable varname
    Set oThisVar = oDoc.Variables(varname)
    End If
  oThisVar.SetContent varcontent, True
  Next

'Finish up
x = MsgBox(varctr & " variables inserted into " & qv_fn & ". OK to Save?", vbOKCancel, "Update_QV_Variables")
If x <> 2 Then
    oDoc.Save
    End If
oDoc.CloseDoc
oQV.Quit
End Sub

★★★

Macro to Copy Cycle Group From One Document to Another

Keeping with the toolkit theme, here's a macro that can copy a cycle group from one QlikView document to another. If you've done much development, you know that it is easy to copy charts and other sheet objects from document to document; but cycle groups cannot be copied and some of them represent a lot of work to completely replicate in another document.

Copy this macro code into the module area of your toolkit document and create a button to execute it. When the macro is executed it will ask for the pathname to the source document containing the cycle group to be copied. Then it will ask for the cycle group name (remember that upper and lower case is important), and then the pathname to the document that the cycle group will be copied into. If either document requires a username or password QlikView will ask for those as the macro opens the documents. As always, make copies of your documents before testing a macro like this.

Here's the macro code you can copy and paste:


Sub Copy_Cycle_Group_Doc_to_Doc
'------------------------------
' /* Copy Cycle Group From One QV Document to Another */
' /* QlikView Maven, March 2013 */
Dim objQV, objSource, objDest, objSourceVar, objDestVar 
Dim objVars, varcontent, objTempVar, varname, i, varctr
'initialize
fn_source=inputbox("Enter pathname to source document containing the Cycle Group you want to copy:","Enter Source Document Pathname")
if trim(fn_source)="" then 'no entry or cancel
  exit sub
  end if

CGName=inputbox("Enter name of Cycle Group to copy:","Enter Cycle Group Name")
if trim(CGName)="" then 'no entry or cancel
  exit sub
  end if
fn_dest=inputbox("Enter pathname to destination document where the Cycle Group is to be inserted:","Enter Destination Document Pathname")
if trim(fn_dest)="" then 'no entry or cancel
  exit sub
  end if

Set objSource=Application.OpenDoc(fn_source)
Set objDest=Application.OpenDoc(fn_dest) 
Set SrcGrp = objSource.GetGroup(CGName)
Set SrcProps = SrcGrp.GetProperties
Set NewGrp = objDest.GetGroup(CGName)
If NewGrp Is Nothing Then 'check if cycle group exists in dest document
  Set NewGrp = objDest.CreateGroup(CGName) 'doesn't exist so create it
Else
  Set NewProps = NewGrp.GetProperties 'does exist so remove old definition
  for i = 1 to NewProps.FieldDefs.Count
    NewGrp.RemoveField 0
    Next
  End If
Set NewProps = NewGrp.GetProperties
'copy elements of cycle group from source to dest
NewProps.FieldDefs.CopyFrom SrcProps.FieldDefs
NewProps.IsCyclic = SrcProps.IsCyclic
NewProps.Labels = SrcProps.Labels
NewProps.Present = SrcProps.Present
NewProps.SortCriterias.CopyFrom SrcProps.SortCriterias
NewGrp.SetProperties NewProps

'we're done, show a message and close down
x=msgbox(CGName&" copied from "&fn_source&" to "&fn_dest&"."&chr(13)&"OK to Save?",vbOKCancel,"Copy_Cycle_Group_Doc_to_Doc")
if x=vbOK then
    ObjDest.Save
    End if
objDest.CloseDoc 'comment out this line if you'd like dest document to stay open for examination
objSource.CloseDoc
set objSource=nothing
set objDest=nothing
set SrcGrp=nothing
set SrcProps=nothing
set NewGrp=nothing
set NewProps=nothing 
End Sub  'end of Copy_Cycle_Group_Doc_to_Doc

★★★

More on Copy Variables From One Document to Another

Here's an alternate version of that macro from last time. This macro can be copied into the module area of a QlikView document (I call mine tookit.qvw) and, when executed, it will copy variables from an external document to another external document. I find this one handier for the kind of work I do.

This document will ask you for the pathname to a source QlikView document containing the variables you want to copy. Then it will ask for the pathname to a destination document that will have the copied variables inserted. If either of the documents requires a password you will be asked for the password as the macro opens the documents. There's no error checking but the most common error for this macro is entering an invalid pathname. After the variables are copied, the macro will ask if it is OK to save the destination document. Remember, as always, to make a backup copy of your documents before testing macros like this.

Here's the code that you can copy and paste into your own toolkit document:


Sub Copy_Variables_Doc_to_Doc
'----------------------------
' /* Copy Variables From One QV Document to Another */
' /* QlikView Maven, February 2013 */
Dim objQV, objSource, objDest, objSourceVar, objDestVar 
Dim objVars, varcontent, objTempVar, varname, i, varctr
'initialize
fn_source=inputbox("Enter pathname to source document containing the variables you want to copy:","Enter Source Document Pathname")
if trim(fn_source)="" then 'no entry or cancel
  exit sub
  end if
Set objSource=Application.OpenDoc(fn_source) 

fn_dest=inputbox("Enter pathname to destination document where variables are to be inserted:","Enter Destination Document Pathname")
if trim(fn_dest)="" then 'no entry or cancel
  exit sub
  end if
Set objDest=Application.OpenDoc(fn_dest) 

'Loop through the variables
set objVars=objSource.GetVariableDescriptions
varctr=0
for i = 0 to objVars.Count - 1
  varctr=varctr+1
  set objTempVar = objVars.Item(i)
  varname=Trim(objTempVar.Name) 
  Set objSourceVar=objSource.Variables(varname) 
  varcontent=objSourceVar.GetRawContent
  'display the variable to check on progress if needed
  'msgbox(varname & " = " & varcontent)

  'update the value of variable in destination document
  Set objDestVar=objDest.Variables(varname)
  If objDestVar is nothing then
    'must need to create variable
    objDest.CreateVariable varname
    Set objDestVar=objDest.Variables(varname)    
    End If

  objDestVar.SetContent varcontent,true

next 'end of loop 

'we're done, show a message and close down
x=msgbox(varctr&" variables copied from "&fn_source&" to "&fn_dest&"."&chr(13)&"OK to Save?",vbOKCancel,"Copy_Variables_Doc_to_Doc")
if x=vbOK then
    ObjDest.Save
    End if
objDest.CloseDoc 'comment out this line if you'd like dest document to stay open
objSource.CloseDoc
set objSource=nothing
set objDest=nothing
set objVars=nothing
set objTempVar=nothing
set objSourceVar=nothing
set objDestVar=nothing
End Sub  'end of Copy_Variables_Doc_to_Doc


★★★

Macro to Copy Variables In From Another Document

My preference when I need to copy variables from one QlikView document to another is to use a small Windows script to do the copying (I wrote about it in 2008 - search on "Copying Variables From One QlikView Document to Another"). But, I know there are some people who can't use a Windows script because of corporate or I.T. rules. Here's a QlikView macro that you can insert into your module area that will copy in all of the variables from another document. 

This macro has you entering a pathname for the document into an input box. For anyone in the advanced class you can replace that with a browse-for-file feature if you think it is worth the time (I wrote about it - search on "Browse-for-File Macro Button"). In the next blog posting ("More on Copy Variables From Document to Another") I show a macro that can be installed once in a toolkit document and it will copy variables from a specified external document to another external document.

Here's the macro code. Try it out - you can copy and paste the text you see in this blog. Remember, as always, to make a backup copy of your document before testing macros like this. To run the macro, create a button for it or for one-time use just click the Test button in the module editor. There's no error checking but the most common error for this macro is entering an invalid pathname. If your source document requires a password then you will be asked for the password when the macro opens the source document.

Sub Copy_Variables
' /* Copy Variables In From Another QV Document */
' /* QlikView Maven, February 2013 */
Dim objQV, objSource, objDest, objSourceVar, objDestVar 
Dim objVars, varcontent, objTempVar, varname, i, varctr
'initialize
fn=inputbox("Enter pathname to source document containing variables you want to copy:","Enter Pathname for Source Document")
if trim(fn)="" then 'no entry or cancel
  exit sub
  end if
Set objSource=Application.OpenDoc(fn) 
set objVars=objSource.GetVariableDescriptions
varctr=0

'Loop through the variables
for i = 0 to objVars.Count - 1
  varctr=varctr+1
  set objTempVar = objVars.Item(i)
  varname=Trim(objTempVar.Name) 
  Set objSourceVar=objSource.Variables(varname) 
  varcontent=objSourceVar.GetRawContent
  'display the variable to check on progress if needed
  'msgbox(varname & " = " & varcontent)

  'update the value of variable in current document
  Set objDestVar=ActiveDocument.Variables(varname)
  If objDestVar is nothing then
    'must need to create variable
    ActiveDocument.CreateVariable varname
    Set objDestVar=ActiveDocument.Variables(varname)
    End If

  objDestVar.SetContent varcontent,true

next 'end of loop 

'we're done, show a message and close down
msgbox varctr&" variables copied from "&fn
objSource.CloseDoc
set objSource=nothing
set objVars=nothing
set objTempVar=nothing
set objSourceVar=nothing
set objDestVar=nothing
End Sub  'end of Copy_Variables

★★★

Make a Copy of a Cycle Group

I sometimes build cycle groups with more than a dozen calculated dimension expressions and labels that are text expressions. It used to irritate me when I needed to make copies of those cycle groups (like when I need a very similar group for a chart with two or more dimensions or a separate group for a graph which may be a little different from a group used for a straight table). Copying the cycle group was a time-consuming chore going back and forth copying and pasting.

Here's a macro you can copy into the module code of a QlikView document that will allow you to easily make copies of a cycle group. I got the code for this macro originally on QlikCommunity and modified it to use input boxes to ask for the cycle group names. It has been been very useful - I usually just copy it into a document temporarily when I need to make duplicate cycle groups and remove the macro afterwards. As always, you should make a backup copy of your document before making major updates or running macros that do updates.

Sub RunCopyCycleGroup
'--------------------
CGName=inputbox("Enter name of cycle group to copy:")
if trim(CGName)="" then 'no entry or cancel
  exit sub
  end if
NewName=inputbox("Enter name for copy of "&CGName&":")
if trim(NewName)="" then 'no entry or cancel
  exit sub
  end if
Call CopyCycleGroup(CGName, NewName)
End Sub
'
Sub CopyCycleGroup(CGName, NewName)
'-----------------------------------
' /* Copy cycle group. OK if NewName exists. */
Set SrcGrp = ActiveDocument.GetGroup(CGName)
Set SrcProps = SrcGrp.GetProperties
Set NewGrp = ActiveDocument.GetGroup(NewName)
If NewGrp Is Nothing Then
  Set NewGrp = ActiveDocument.CreateGroup(NewName)
Else
  Set NewProps = NewGrp.GetProperties
  for i = 1 to NewProps.FieldDefs.Count
    NewGrp.RemoveField 0
    Next
  End If
Set NewProps = NewGrp.GetProperties
NewProps.FieldDefs.CopyFrom SrcProps.FieldDefs
NewProps.IsCyclic = SrcProps.IsCyclic
NewProps.Labels = SrcProps.Labels
NewProps.Present = SrcProps.Present
NewProps.SortCriterias.CopyFrom SrcProps.SortCriterias
NewGrp.SetProperties NewProps
msgbox CGName&" copied to "&NewName
End Sub

I have also posted a macro that will copy a cycle group from one document to another, separate document (search on "Macro to Copy a Cycle Group From One Document to Another").

★★★

Macro Copies Value From a Chart into a Variable

Here's an example of a method that will allow a QlikView macro to look at a value inside a sheet object like a straight table and put the value into a variable. Then the variable can be used in an expression like a text box or small chart. Most of the time, something like this could be accomplished with an expression in the variable that produces the same value as what you see in the chart. But a macro would be useful when duplicating the chart value with an expression would be difficult because of varying sort sequences or calculated dimensions in the chart, etc.

This is the macro code:

Sub Get_XTOTAL
 Application.WaitForIdle
 set cell=ActiveDocument.GetSheetObject("CH45").GetCell(1,7)
 ActiveDocument.Variables("vXTOTAL").SetContent cell.text, True 
 set cell=nothing
End Sub

This example macro is extracting a value from the totals row of the chart where the object ID is CH45 (look in the General tab of the chart properties to find the object ID). The number arguments for the GetCell method are row and column; row 1 and column 7 in this example.
Then the macro updates the value in the document variable named vXTOTAL. When written as in this example, the variable must already exist, so create the variable before executing the macro.
When I use a macro like this, I set up a trigger to execute the macro as needed, usually when a selection changes.

★ ★ ★

Select Top 4 Values On Open

I was working on a document that contained a lot of data and the performance when the document was opened or when the user moved from tab to tab was a little slow. On my own laptop when I save documents like that I usually make a selection of a small amount of data so that later when I open the document again it will open quickly. I wanted to do something like that for the users -- some kind of selection that would happen automatically when the document is opened, but it had to be a selection of data that would be useful and easy for the users to understand.

The data in the document was organized by a date field named POSTDATE so I decided that I would like the document to automatically select the most recent four POSTDATE values when it was opened. Most users look in the document for the most recent data and anyone who wanted to review older data could easily add older dates to the selection through the multibox.

The method I used is to create a trigger that would fire when the document is opened. The trigger would do a Pareto Select. Now, I'm not saying that this is the best way to accomplish the selection and it certainly is not the only way, but it was easy and didn't involve writing macro code.

Pareto Select is used to select a percentage of values in a field based on ranking of a value or expression. It is commonly used select things like the top 20% best selling products. In this case I wanted to use it to select not a fixed percentage but the four most recent POSTDATE date values. This is what I did:

From the menu choose Settings->Document Properties and choose the Triggers tab.
In the document event triggers click the Add Action button for the OnOpen event.
Click Add Action and in the Add Action dialog choose Action Type=Selection and Action=Pareto Select then click OK.
In the Actions dialog type POSTDATE into both the Field and Selection boxes. POSTDATE is not only the field I want to select but also the expression value I want ranked.
Then, in the Percentage box enter =(4/COUNT(DISTINCT POSTDATE))*100
And click OK. Save the document and the next time it is opened it will automatically select the four most recent POSTDATE values.

A similar method could be used to do automatic selection of any number of top values of any field. Pareto Select does its selection based on the current Possible values. If you wanted to ignore the current selection and select a percentage of all values then you should add an extra action to the trigger to clear the field before making the Pareto Selection.

Testing trigger actions is most easily done by defining the same actions for a button and then you can test the action by clicking the button. When you're done testing, remove the button and make sure the actions are associated with the proper event.

= = = = = = = = = = = = = = = = = = = = = = = = = =
Author's Note 6Dec2011:
This method didn't work the way I wanted when the document already included selections on the POSTDATE field. I couldn't find any good way to deal with that using the available trigger actions. I eventually replaced the Pareto Select action with a macro action that selects the four most recent POSTDATE values. The macro code looks like this:
sub Top_4_POSTDATE
set fd=ActiveDocument.GetField("POSTDATE")
fd.Clear
fd.TopSelect "=if(aggr({1} rank(POSTDATE),POSTDATE)<=4,POSTDATE)",4
set fd=nothing
end sub

* * *

Browse-for-File Macro Button

Several times I’ve created QlikView reports that must read data from a file but the name of the file can vary and must be supplied by the report user. Often this is a file that will be used by the loadscript so the user first clicks a macro button that runs a browse-for-file dialog and then the user clicks the standard Reload button.

Here’s an example of a macro that borrows the browse file dialog method from the Microsoft Excel object model. This macro should work on any PC that has MS Excel installed (it does not actually start Excel). You can copy the code from here and paste it into your QlikView report module code:

Sub Browse_for_File
'------------------
' Get the file pathname
Set oXL=CreateObject("Excel.Application")
f_name=oXL.GetOpenFilename("All Files (*.*),*.*",,"Select file",False)
If f_name="False" then
  'user cancelled out of dialog box
  Set oXL=nothing
  Exit sub
End If
'store file pathname in the file_pathname variable
ret=ActiveDocument.GetVariable("file_pathname").SetContent(f_name,false)
Set oXL=nothing
End Sub

This macro stores the file pathname for the file selected by the report user into a document variable named file_pathname (create the new document variable before running the macro). When I create a report that uses this macro I usually also put an input box in the report that shows the file pathname stored in the variable and allows the user an alternative method for specifying the pathname.

A QlikView document example of this technique, named BIG_Text_File.qvw, is available here – http://finmagic.0catch.com/ Sorry about the ads and popups – just click on the specific file download.

See a Title Tab When Report Opens

Normally, the active sheet or tab a person sees when they open a QlikView document is the sheet that was active when the document was last saved. So, the easiest way to influence which sheet a person sees when they open the document is just to be careful which sheet is active when the document is saved and closed.

You can also control which sheet or tab a person sees when they open a QlikView report with a small macro. We often do this when a report contains a title tab. The title tab contains information about the report. It would show the report title, of course, and could also show the date the data was loaded, databases used, contact info for the author or support person, security label, etc. It is also a great place for a company logo or other identifying graphics.

Step one is to build the tab you want users to see when they open the report.

Second, create a small macro (Settings->Document Properties...->Macros). For example, if the name of the title tab is Title_Tab then the macro would look like this:

Sub Activate_Title_Tab
'---------------------
ActiveDocument.Sheets("Title_Tab").Activate
End sub

Then, associate the macro with the OnOpen document event trigger and click OK.

Now you have a macro that is executed whenever the document is opened and the macro activates the tab that you want the user to see first. The Activate_Title_Tab macro is also available to be associated with a button or other events or executed by other macros when it is appropriate to switch to a particular sheet or tab.

Checking for Loadscript Errors from the Macro code

If you are using the macro code Reload method to load data into your QlikView report you can also easily check for any loading errors. Here's an example of how that could be done. Add a small subroutine to the macro code that can check the system variable for error count like this:

Sub Check_for_errors()
'---------------------
'See if any load script errors occurred and report them
alarm_flag = 0
Set objvar = ActiveDocument.Variables("ScriptErrorCount")
If trim(objvar.GetContent.String) = "0" then
  'No error - hooray
  Exit Sub
  End If
alarm_flag = 1
'Here you can add any other error notification
End Sub

And then, in the subroutine that reloads the document code add an IF statement after the .Reload method something like this:

ActiveDocument.Reload
Check_for_errors 'call Check_for_errors subroutine
If alarm_flag = 1 then 'an error happened in the loadscript
  ActiveDocument.CloseDoc
  ActiveDocument.GetApplication.Quit
  '***Exit Function or Exit Sub
  End If

In the reports I've developed there is error checking every step of the way and any error causes a notification file to be written out. If you have an email client program on the computer where the report is reloading then consider using an email notification to the report users or to a technical support person. I've sometimes sent a small notification email message to my cell phone (My favorite message is "Lassie, go for help!") so that I can be notified immediately with the bad news.

VBScript in Loadscript and Module

The language used in both the QlikView loadscript and in the module code (macros) is modified VBScript. You can easily find information about VBScript commands and syntax through Google. For example, to find information about msgbox prompts and syntax enter this in Google:
vbscript tutorial msgbox