Utility That Can Extract and Show All of the Variables from Your QlikView Document

Some of our project teams at work are required to use a standard QlikView document for several clients and then use a client-specific loadscript .qvs file to configure custom variables for each client. The loadscript files define the variables with a set of Let statements and I often get questions about the syntax for the Let statements. Some of the variables define complex expressions and they include dollar signs and functions and quotes and some definitions use multiple lines so the Let statement syntax can get tricky.

I recommend that people do their development first so that they have a working document with all of the desired features and variables that are required. Then, they can use a QlikView document I wrote to be used as a utility program to help people with managing their variables.

You can download a working copy of the utility program, named List_Out_Variables.qvw, by clicking HERE

The utility program has a tab with some text instructions and an input box where people type in the complete pathname to the custom .qvw document that they’ve been working on. Then, they do a reload of the utility program and it extracts all of the variables from the custom document.

The input box stores the pathname in a variable named vDocumentPath. Part of the loadscript in the utility program looks like this:

VARIABLES:
LOAD
  [Name] as VARIABLE_NAME,
  [RawValue] as VARIABLE_VALUE
FROM  $(vDocumentPath) (XmlSimple, Table is [DocumentSummary/VariableDescription]);

That loadscript code extracts the variables from the document and makes the variable names and values available to be shown in charts. (I borrowed the loadscript code from a Rob Wunderlich comment I found in QlikCommunity.)

The second tab on the utility is pretty simple, it is table box showing the VARIABLE_NAME and VARIABLE_VALUE fields along with list boxes for those two fields so that people can select the variables they are interested in. The utility is pretty useful right there as a way to analyze variables, sort them, print them, copy and paste to other documents, etc.

There is a third tab with a chart that helps people with the Let statement syntax that I mentioned. The chart on that tab uses VARIABLE_NAME as the dimension but I found it helps to have the dimension column hidden. Then, there is one expression that converts the VARIABLE_VALUE into a Let statement. The expression looks like this:

'Let '&VARIABLE_NAME&'='&chr(39) &
Replace(Replace(Replace(VARIABLE_VALUE,vQUOTE,vQR),vDOLLAR,vDR),vCR,vCRR)
& chr(39)&';'

An example of how that looks in the actual chart is:

Let MoneyFormat=''&chr(36)&'#,##0.00;('&chr(36)&'#,##0.00)';

To make that work I had to define a few variables in the utility program. It made that text expression much simpler to write. The variables in the utility program are:

vDollar is a variable containing a single dollar sign

vQuote is a variable containing a single quote

vCR is a variable containing a carriage return (really simple, when defining this one just put your cursor in the variable definition window and click your Enter key).

vQR is  '&chr(39)&'

vDR is  '&chr(36)&'

vCRR is  '&chr(13)&'      

If you download a copy of the utility program then you don't have to worry about any of that coding as it is already done for you.

After loading a set of variables extracted from your special document, my instructions tell people to make their selections and then, from the chart, they can right-click and export the chart or copy the data to the Windows clipboard so it can be pasted into a .qvs file or pasted into a loadscript or pasted in some other document.

.

Having an easy method to define a set of variables as Let statements can be used to copy variables from document to document or to accompany a version control process where variables and their values need to be stored as text in a way that allows them to be analyzed or restored later.

Some folks familiar with this blog may remember that several years ago I wrote about an Excel spreadsheet containing a macro that could extract QlikView variable names and values. It worked great but is no longer useful in our corporate environment as the corporate anti-malware software considers it a “suspicious program” and refuses to let it run. The List_Out_Variables.qvw utility is a good replacement.

  ★★★

Have you got a function similar to SQL NVL ?

I work with a group of people who are all experts in SQL. They all also have varying levels of technical skills and QlikView skills. One question I get frequently is whether QlikView has a function like the SQL NVL function.

For those of you not familiar with relational database SQL language, the NVL function takes two arguments: a field name or expression and a default value to be returned if the first argument is null. For example, a SQL database query may include the function like this:

  NVL(ORD_DISCOUNT, 0) AS ORD_DISCOUNT

And that would tell the SQL processor to look at the value of ORD_DISCOUNT and if it is null then return 0 as the function value otherwise return ORD_DISCOUNT.

When people ask about achieving the same thing with QlikView, I usually start by telling them that they can code an “if” statement like this:

  If(IsNull(ORD_DISCOUNT), 0, ORD_DISCOUNT)

and then I explain that there is a built-in QlikView function that can be used similar to NVL as long as the field you are checking is supposed to be numeric. It is the Alt function.

The Alt function accepts any number of arguments and looks at each of them going from left to right and returns whichever one is a valid number. So, repeating our example, if ORD_DISCOUNT is null then the following function will return a zero but if ORD_DISCOUNT is a valid number then the function returns ORD_DISCOUNT:

  Alt(ORD_DISCOUNT, 0)   

The Alt function will treat the last or rightmost argument as an “else” condition and return that value if none of the preceding arguments are numeric. The rightmost value need not be numeric so you could code something like this:

  Alt(ORD_DISCOUNT, 'Discount is missing')

The Alt function may be used in the loadscript code or in chart expressions.

If you are interested, the QlikView Help (search in Help for Conditional Functions) shows an interesting example for how the Alt function can be used to identify a date when the date value may be any one of several different date formats.  

  ★★★

MonthsBetween Function

A few days ago I needed a calculation in a loadscript that would provide the number of months between two dates. This kind of calculation comes up sometimes with supply chain planning data when the difference between the month a sales forecast is developed and the month that the sales forecast is meant to be applied, sometimes called a monthly lag, can be used to help measure forecast accuracy. QlikView provides a number of date functions but not one that can calculate the number of months between two dates.

I decided to make my own MonthsBetween date function using a document variable with input arguments. A variable like that can be used like a user defined function. I created a variable in the QlikView document and named it MonthsBetween. This is what I typed for the content of the variable (you, of course, can copy and paste from this blog post):
((Year($2)*12 + Month($2)) - (Year($1)*12 + Month($1)))

The $1 and $2 get replaced by two input arguments (which some people might call parameters) when the variable is used. The arguments must be dates. My loadscript (I was loading data from an Oracle database table) looked something like this:

BUDGET_HIST:
Load BUSINESS_UNIT,
BUDGET_AMOUNT,
CREATE_DATE,
APPLY_DATE,
$(MonthsBetween(CREATE_DATE, APPLY_DATE)) AS MONTH_LAG;
Select * from BUDGET_HIST_TABLE;

For example, if the CREATE_DATE is 1-APRIL-2016 and the APPLY_DATE is 1-JULY-2016 then MONTH_LAG is 3. If the second date is later than the first date then the result will be a positive integer number of months. If the two dates are from the same month then the result is a zero. For my data, the APPLY_DATE is always later than or equal to the CREATE_DATE so MONTH_LAG is always 0 or greater. If CREATE_DATE was later than APPLY_DATE then MONTH_LAG would be a negative number.

The calculation is nothing special -- it must have been re-created thousands of times. But, putting it into a variable makes the calculation easy to use in many places and easy to share. Test it out in a chart expression or advanced search expression. 

When a variable contains expression code like that and the variable might be used as part of a larger expression in a chart, then it is a good idea to enclose the entire variable contents within parentheses as I did above. It helps prevent QlikView from processing the operators in the larger expression in the wrong order.


★★★  

Inline Load with Where Clause

You wouldn’t ordinarily use a Where clause with an inline load… after all, if you didn’t want all of the rows why bother typing them? But, as an example, I had a project two weeks ago where an inline load with a Where clause helped solve a problem. It was a QlikView document that could load data from any one of several dozen companies. The data included a company code but not the company name or any other friendly details for me to use in the document.

I coded an inline load with a row for each of the possible companies. In the loadscript, after loading the company data, I coded an extract of the company code from the data and stored it in a variable named var_cmp_code.

The inline load, simplified here, looked something like this:

COMPANY_INFO:
Load * inline
[cmp_code, company_name, company_industry
BA, Boeing, Aerospace
CCC, Chesterfield Coal Co, Mining and Minerals
LPL, London Pastry Ltd, Stale Biscuits]
Where cmp_code = '$(var_cmp_code)';

Since each load of the document is loading data for one company, the Where clause on the inline load makes it load one row only and each field name, therefore, has only one value. That allowed me to use company_name or any of the other field names in chart titles and text boxes. For example, I used a text expression in a chart title like this:  ='Future Health Care Risk by Year for ' & company_name

Remember that an inline load is like any other load except that the data is typed directly into the loadscript instead of being loaded from a separate file or database. In addition to a Where clause, you could code a While, Autogenerate, mapping table, Concatenate, Qualify, or the various types of joins available in a loadscript.

★★★

Oracle Hint in the Loadscript

People who load data from an Oracle database into QlikView may sometimes want to use an Oracle hint in the SQL query code. An Oracle hint tells Oracle about how you would like it to plan and execute the query in order to get faster, more efficient execution.

The Oracle hint syntax is actually a comment embedded in the SQL query. For example, the hint to tell Oracle to use up to four parallel processors might look like this in the first line of the query:
SELECT /*+ parallel(4) */ CUSTOMER, SUM(ORD_QTY) FROM   . . .

If you simply code a query like that in the loadscript you won’t get an error or a warning, the query will return data, but you won’t get the performance improvement you were hoping for either. That’s because QlikView doesn’t ordinarily pass comments through to the SQL processor. Oracle would never see your hint.

What is needed, is for you to tell QlikView not to strip out the comments before passing the query into the SQL processor. Code a line like this before your loadscript table definition:

Set StripComments = 0;

ORDER_TABLE:

SELECT /*+ parallel(4) */ CUSTOMER, SUM(ORD_QTY) FROM   . . .

The StripComments variable is a system variable that controls QlikView behavior. Setting the variable to zero tells QlikView not to strip out the comments. It is a good practice to turn StripComments back on after the query with a line like this:
Set StripComments = 1;

One thing to look out for is any other comments in your code. If you have a regular loadscript comment //(a comment beginning with two slashes) mixed in with your SQL code then it will cause a SQL error when it gets passed in to the SQL processor.

★★★

Loading an Unstructured Text File

This is the start of a series of postings about dealing with unstructured text data. I've used QlikView to help analyze message log files and error files and even program source code – there are some things QlikView can do that Notepad style text editors cannot.
First, we need to load the data. It isn’t always a completely unstructured file like my example, often the text is one or more fields inside a structured file or database.  There are various ways to load a text file; this is what I usually put into the loadscript:
TEXTFILE:
LOAD 
RecNo() as recno,
@1 as rectext
FROM
[$(File_Pathname)]
(txt, no labels, delimiter is \x7, no quotes); 

That code will load each record from the text file into the rectext field. I use a variable for the file pathname because I usually use a loadscript like this with a document where the user can supply a filename through an inputbox. Alternatively, you could add browse-for-file functionality like I wrote about in March 2009 (search on QlikView Maven Browse-for-File Macro Button). Because the data is unstructured and the document objects are meant to be used with unstructured data, the document works well as a general-purpose tool that can be used with many types of data in varying situations. 

I defined the file as though it is delimited and made the specified delimiter an uncommon character I don’t expect to find in the data. 

The record number, loaded as recno, from the input file is a useful piece of data and we’ll be using it in some of the example objects over the next few postings.

★★★

Macro to Copy Variables In From Another Document

My preference when I need to copy variables from one QlikView document to another is to use a small Windows script to do the copying (I wrote about it in 2008 - search on "Copying Variables From One QlikView Document to Another"). But, I know there are some people who can't use a Windows script because of corporate or I.T. rules. Here's a QlikView macro that you can insert into your module area that will copy in all of the variables from another document. 

This macro has you entering a pathname for the document into an input box. For anyone in the advanced class you can replace that with a browse-for-file feature if you think it is worth the time (I wrote about it - search on "Browse-for-File Macro Button"). In the next blog posting ("More on Copy Variables From Document to Another") I show a macro that can be installed once in a toolkit document and it will copy variables from a specified external document to another external document.

Here's the macro code. Try it out - you can copy and paste the text you see in this blog. Remember, as always, to make a backup copy of your document before testing macros like this. To run the macro, create a button for it or for one-time use just click the Test button in the module editor. There's no error checking but the most common error for this macro is entering an invalid pathname. If your source document requires a password then you will be asked for the password when the macro opens the source document.

Sub Copy_Variables
' /* Copy Variables In From Another QV Document */
' /* QlikView Maven, February 2013 */
Dim objQV, objSource, objDest, objSourceVar, objDestVar 
Dim objVars, varcontent, objTempVar, varname, i, varctr
'initialize
fn=inputbox("Enter pathname to source document containing variables you want to copy:","Enter Pathname for Source Document")
if trim(fn)="" then 'no entry or cancel
  exit sub
  end if
Set objSource=Application.OpenDoc(fn) 
set objVars=objSource.GetVariableDescriptions
varctr=0

'Loop through the variables
for i = 0 to objVars.Count - 1
  varctr=varctr+1
  set objTempVar = objVars.Item(i)
  varname=Trim(objTempVar.Name) 
  Set objSourceVar=objSource.Variables(varname) 
  varcontent=objSourceVar.GetRawContent
  'display the variable to check on progress if needed
  'msgbox(varname & " = " & varcontent)

  'update the value of variable in current document
  Set objDestVar=ActiveDocument.Variables(varname)
  If objDestVar is nothing then
    'must need to create variable
    ActiveDocument.CreateVariable varname
    Set objDestVar=ActiveDocument.Variables(varname)
    End If

  objDestVar.SetContent varcontent,true

next 'end of loop 

'we're done, show a message and close down
msgbox varctr&" variables copied from "&fn
objSource.CloseDoc
set objSource=nothing
set objVars=nothing
set objTempVar=nothing
set objSourceVar=nothing
set objDestVar=nothing
End Sub  'end of Copy_Variables

★★★

See a Title Tab When Report Opens

Normally, the active sheet or tab a person sees when they open a QlikView document is the sheet that was active when the document was last saved. So, the easiest way to influence which sheet a person sees when they open the document is just to be careful which sheet is active when the document is saved and closed.

You can also control which sheet or tab a person sees when they open a QlikView report with a small macro. We often do this when a report contains a title tab. The title tab contains information about the report. It would show the report title, of course, and could also show the date the data was loaded, databases used, contact info for the author or support person, security label, etc. It is also a great place for a company logo or other identifying graphics.

Step one is to build the tab you want users to see when they open the report.

Second, create a small macro (Settings->Document Properties...->Macros). For example, if the name of the title tab is Title_Tab then the macro would look like this:

Sub Activate_Title_Tab
'---------------------
ActiveDocument.Sheets("Title_Tab").Activate
End sub

Then, associate the macro with the OnOpen document event trigger and click OK.

Now you have a macro that is executed whenever the document is opened and the macro activates the tab that you want the user to see first. The Activate_Title_Tab macro is also available to be associated with a button or other events or executed by other macros when it is appropriate to switch to a particular sheet or tab.

Copying Variables From One QlikView Document to Another

Today, one of the guys at work was trying to figure out how to transfer all of the document variables from one QlikView document to another. We discussed some of the things we've done in the past: loading variables and values from a spreadsheet or database either through the loadscript or through macro code. We even have an older application that stores variables in a table with loadscript code and then the document is used as a "binary" in another document's loadscript and the second document turns the table values back into variables. But, none of those options seemed to fit the requirement. One obstacle is that we could not figure out an easy way to export all of the variables and values from a document. You can export the expressions from the Expression Overview but the Variable Overview has no similar function.

With some help from Lars at Brait AB (a reader of this blog, Thanks, Lars!) who suggested a way to loop through the variables, I tested the VBScript below that can transfer variables from one document to another. I created a small text file on my laptop named "tbtest.vbs" and typed this code into it:

' /* Test Moving a variable from one QV document to another */

' /* QlikView Maven, Nov 2008 */
Option Explicit

Dim objQV, objSource, objDest, objSourceVar, objDestVar

Dim objVars, varcontent, objTempVar, varname, i

'initialize

Set objQV=CreateObject("QlikTech.QlikView")

Set objSource=objQV.OpenDoc("C:\QVfolder\Source_rpt.qvw")

Set objDest=objQV.OpenDoc("C:\QVfolder\Dest_rpt.qvw")
set objVars = objSource.GetVariableDescriptions

'Loop through the variables

for i = 0 to objVars.Count - 1
  set objTempVar = objVars.Item(i)

  varname=Trim(objTempVar.Name)

  Set objSourceVar=objSource.Variables(varname)

  varcontent=objSourceVar.GetRawContent

  'display the variable to check on progress if needed

  'msgbox(varname & " = " & varcontent)

  'update the value of variable in Destination document

  Set objDestVar=objDest.Variables(varname)

  If objDestVar is nothing then

    'must need to create variable

    objDest.CreateVariable varname

    Set objDestVar=objDest.Variables(varname)

    End If

  objDestVar.SetContent varcontent,true

next 'end of loop

'save Destination document if desired

objDest.Save

'we're done, close down

objSource.CloseDoc

objDest.CloseDoc
objQV.quit
WScript.Quit

In order to run it, I just double-clicked on the tbtest.vbs file on my desktop. You can see the pathname to the two QlikView document files in the code. This script successfully copied the variables and their values from the source to the destination document.

If you omit the part of the script that opens and updates the destination document you could turn it into a script that lists out all of the report variables and their contents -- something that is not available with the regular variable overview.

More you can do with Partial Reload

See the previous posting about removing tables. In the same way that you can enter a Drop Table command to remove a QlikView table from an existing report, you can enter these commands which will affect the existing report already loaded with data:

• To remove a field from every QlikView table in which it appears, enter Drop Column;
  For example, for privacy reasons you might want to get rid of social security number data in your report before sharing it with your team. You might enter:
  Drop Field Customer_SSN;


• To remove a field from specific table(s), for example, you might enter:
  Drop Fields Customer_Name, Customer_SSN from CUST_MSTR_TABLE, ORDER_HDR_TBL;


• To change the name of a column or field in every table in which it appears enter Rename Field; (remember that the field names are how QlikView relates or "joins" the tables together so this one affects that table relationship).
  For example, you might enter:
  Rename Field domestic_custno to North_America_Customer;


• You can change the name of an existing table. For example, you might enter:
  Rename table domestic_sales to North_America_Sales;

All of these commands can also be used in the loadscript during a normal load but they can be especially useful for fixing issues when used with an existing report that is already loaded with data. Remember to put them at the top of the loadscript, add an Exit Script; command and then execute the commands with the Partial Reload. The rename commands can be used with a mapping table if you think you might need more powerful renaming capabilities. Refer to the Help Text or Reference Manual for info about how to use a mapping table for renaming fields and tables.

How to remove an existing table

For this first blog entry we'll talk about something simple although maybe not a beginner's task. How do you remove a QlikView table from an existing report that is already loaded with data? There's a variety of reasons you might do something like that: to remove a large table and reduce the size of the report, maybe the data in one table is incorrect or causes incorrect results in the charts, or maybe you intend to share the report with Marcy in Finance and she just doesn't need to see the sales forecast. Whatever the reason, it is fairly easy to remove a table.

First, make a copy of the report you are going to change so that you can restore the report in case something goes wrong or you make a mistake (it happens). Open the report, then select File->Edit Script from the menu. It will open the loadscript edit window. Insert a few blank lines at the very top. Now type in a drop table command for the table you want to remove. For example, if you want to remove the NEW_ORDERS table then type in
DROP TABLE NEW_ORDERS;
EXIT SCRIPT;

Don't forget the semi-colons at the end of those lines. It might look something like the picture up above.

After you type the new lines in the loadscript, review the lines to make sure everything is spelled correctly and click on the OK button to close the edit loadscript window and then select File->Partial Reload from the menu. The partial reload choice will execute your new loadscript lines down to the Exit script line and then stop. The NEW_ORDERS table has now been removed from the report. If you ever expect to reload the report again you should edit the loadscript again and remove the new lines you just inserted. Then click OK to close the edit loadscript window and Save the report (or do a Save As to save the report under a different file name).