QlikView Exists Function

One QlikView function I use often in the loadscript is the Exists function. You give it a value or expression and the name of a field and it tells you if the value already has been loaded into the field. The function returns a True or False so it can be used in the Where clause of a Load statement or in an If statement. There are a lot of uses for the function and below are three examples that I’ve used recently. For the first example, I use it near the end of the loadscript when I am loading master data to match data already loaded in previous tables. Here’s what the code might look like when loading a product description to go along with previously loaded product code values:

PRODUCT_MASTER:
Load
PRODUCT_CODE,
PRODUCT
Where Exists(PRODUCT_CODE);
SQL SELECT
PRODUCT_CODE,
PRODUCT_CODE' - 'DESCRIPTION AS PRODUCT
FROM PRODUCT_MASTER;

That piece of code will only load product descriptions for product codes that were already loaded in previous tables. Note that the Exists function is a QlikView function so it is used as part of the QlikView Load statement and not as part of the SQL statement. Only one argument for the function is used in this case since the field name and the value to be checked are the same.

A second example of a place I use the Exists function is when I load temporary tables with special field names and values that fit a particular type or grouping so I can use the values to differentiate values in other fields I’m loading. Here’s an example: I load product codes into a special field that will contain only product codes for products made with 100% recycled materials:

TEMP_RECYCLE_PRODUCTS:
Load
PRODUCT_CODE as RECYCLE_PRODUCT
Resident PRODUCT_MASTER
Where CONTENT='100 POST-CONSUMER RECYCLE';

Now, I can use that field name, RECYCLE_PRODUCT, with the Exists function as I load or process other data. This technique is especially useful when the data is coming from different sources where something like a SQL join is not available. I might use it like this when loading other data:

GREEN_SALES:
//Sales of product made from recycled materials
Load
CUSTOMER,
SALES_DATE,
QUANTITY
Resident 2009_SALES_DETAIL
Where Exists(RECYCLE_PRODUCT,PRODUCT_CODE);

Drop table TEMP_RECYCLE_PRODUCTS;

At the end of the loadscript or when it is no longer needed you should drop temporary tables so that they don't unnecessarily use up memory or create unneeded associations.

In this third example, I use the Exists function to load only the first of a set of identical values as they are read in the input data. For example, this piece of code in the loadscript will only load data for the first time an error message appears in an error message log file. The input from the database is sorted into timestamp order and the Exists function in the Load statement checks to see if the specific error message value has already been loaded:

ERROR_HISTORY:
Load
ERROR_TIMESTAMP,
ERROR_MESSAGE
Where Not Exists(ERROR_MESSAGE);
SQL SELECT
ERROR_TIMESTAMP,
ERROR_MESSAGE
FROM MESSAGES_LOG
ORDER BY ERROR_TIMESTAMP;

Benefits of Concatenating a Code and Description

In the typical business report in a corporate environment you need to include quite a few selections either through a list box or multi-box. There may be selections for product, customer, sales executive, division, channel, product type, country, fiscal period, etc. You probably have codes for those things in your corporate databases and, if you’re lucky, you also have descriptions that go along with the codes.

When we build a new QlikView report we design the loadscript to load not just the codes but also the concatenation of the code and the corresponding description separated by a hyphen. That part of the loadscript code might look like this:
PRODUCT_MASTER:
SQL SELECT
  PRODUCT_CODE,
  PRODUCT_CODE||' - '||DESCRIPTION AS PRODUCT
FROM PRODUCT_MASTER;

Then, we use the PRODUCT field in list boxes, multi-boxes and in the chart dimensions. The reason this idea is so valuable is because of QlikView’s behavior when you are making a selection: if you click on the title bar of a list box or on the down arrow of a multi-box and then start typing, the visible list of items changes to show only the ones that match what you have typed so far.

For example, if you click the down-arrow on the multi-box selection for our example field, PRODUCT; and then start typing baby -- all of the products that contain the word “baby” in the product description move to the top of the list high-lighted with a white background. Keep typing, baby lotion… and now only the products whose names contains the words “baby lotion” are on the top of the list. At that point you can either press the Enter key which will select all of the high-lighted products or you can click on an individual product with your mouse. If that field only contained product codes then you would be limited to typing in a subset of the product code – which you can still do, of course, with the concatenated field values but the extra value of being able to type in a partial description to help with the selection is extremely useful.

A description for a code value is so useful that we have sometimes loaded the description data from a spreadsheet when it isn’t available on the corporate database. Then it is joined and concatenated to the code value in the loadscript.

Using the concatenated code and description as a dimension in the charts also works well. The field still sorts by product code since that is the leftmost part of the field value. Often we shorten the field width so that just the product code is showing on the screen but moving your cursor over the field reveals the complete description.

Error: fetched column value was truncated

I got an error yesterday on a QlikView document that had reloaded dozens of times before. It appeared to be an Oracle SQL error. The error message that appeared in the pop-up box during loading said, SQL Error:[Oracle][ODBC][Ora]ORA-01406:fetched column value was truncated

Despite appearing like a SQL error, the piece of SQL code didn’t cause any errors when submitted through Oracle SQL*Plus. It only caused an error when used to load a table in the QlikView document. I was using QlikView version 8.50.6206.5.

With a little trial-and-error I narrowed down the source of the problem to a single database column, named QTY, which was defined in the Oracle database as a FLOAT. Since this document had reloaded successfully in the past, it must be the new data stored in the table that is causing the problem now.

In any case, I was able to make the document load and satisfy the reporting requirements by simply changing the SQL a little from SQL SELECT QTY… to SQL SELECT ROUND(QTY,2)… After this change, the document reloaded with no problem.

I’m hoping this blog entry might help someone fix the problem if it happens to them. And, inasmuch as this blog also serves as my own QlikView notes it might serve as a reminder to myself when it happens again in one of my documents.

(Note the comment describing another experience with this error)

Load a Table With All of the Values for a Field

An idea for developers of large reports--

Sometimes in the loadscript you need to build a table that contains all of the previously loaded values from the tables that have already been loaded. This situation comes up when a concatenated key has been used in the tables. For example, you might have a report that loads data into several tables from different sources but you’ve loaded each of the tables so that it has a key field comprised of product_code, customer_number, ship_location, and ship_date; all concatenated together but separated with an underscore character. This technique is sometimes used in large reports in order to reduce memory requirements by minimizing the number of synthetic keys that QlikView must create. Each of the large tables shares only one field, the concatenated key field, instead of sharing multiple fields and causing QlikView to build the synthetic keys.

At the bottom of the loadscript you must collect all of the concatenated key values and explode them into the individual fields so that QlikView can build the associations. If the concatenated key from our example is named CCKEY then code like this would load all of the existing CCKEY values:

ALL_CCKEY_TABLE:
Load
FieldValue('CCKEY',IterNo()) as CCKEY
AutoGenerate(1)
While not Isnull(FieldValue('CCKEY',IterNo()));

That will create a table of all CCKEY values. Now add the part that splits out the fields based on the underscore separator character:

Left join load
CCKEY,
subfield(CCKEY,'_',1) as product_code,
subfield(CCKEY,'_',2) as customer_number,
subfield(CCKEY,'_',3) as ship_location,
subfield(CCKEY,'_',4) as ship_date
Resident ALL_CCKEY_TABLE;

A table like this loads very fast since all of the data is already available in memory.

Archive Data

We sometimes use QlikView as a method for capturing and saving data for archive purposes. These reports aren’t normally used unless someone needs to review historic data or research data to answer a question. For example, if we have a project where we might need to check on what the product master looked like six weeks ago (most companies only care about current master data and they don’t keep track of historic master data), then we might set up a daily job that loads a QlikView document with the product master and any associated data. Then, the document is simply saved in case it is needed. We usually give the document file a filename containing the date to help organize the files.

This advantage of this method over some kind of data dump file is that the method for reviewing the data (QlikView) is automatically part of the process. Anyone who needs to review the data need only click on one of the files in the archive folder. The data in QlikView qvw files is stored in compressed form so no further zipping is needed to conserve disk space.

Adding an Image to the Document

You can easily add visual interest to your QlikView document with a few lines or colored boxes or graphic elements. Add lines of any thickness or color by selecting from the menu Layout->New Sheet Object->Line/Arrow Object... The object properties are used to adjust the color and appearance of the line or arrow. You can drag the line around your document to get the most logical or appealing placement.

For colored boxes that stand out from the tab or sheet background insert a text object (Layout->New Sheet Object->Text Object). In the object properties choose a background color that works well with the sheet background. Drag the text object around and adjust the size with your mouse. Whether you actually have text in your text object is up to you. The text objects can be overlaid or layered with other sheet objects like charts. This is a common way to add a legend or explanation of the data that will appear alongside a chart. The Layout tab in the object properties has a Layers setting that will help with overlapping objects. The transparency setting can also help with layered objects. Don't hesitate to click the Help button for more information if you see something that you don't understand.

The background of the tab or sheet can be changed by right-clicking on the top of the sheet where the tab label is. Then select Sheet Properties and look for the controls that adjust background color or let you add an image. A background photograph is often too busy for a sheet containing chart objects but it may be perfect for a title sheet. The background and colors of a well-designed website might give you some ideas for how to design the sheets of your document.

The text object can show a diagram or picture. This is an easy way to add a corporate or department or project logo to the document. When you add the text object look at the General tab of the object properties. In the Background area click the Image button and then the Change button (btw- note how QlikView uses lines and boxes to group the logical elements of the properties window… a few lines can make the information on your document easier to understand too). Then you can select a jpg or png file from your computer and insert it into the document. Try it out! Even if you are designing a business document there's no need to be boring. Insert a cartoon image of your boss... but before the document is seen by company management you should make sure the image is respectable and that you or your company own the rights. Consider this method for adding a diagram to your document or adding photos of products or a map of locations.

Text objects can be set up so that they appear or are invisible based on the data. On the text object Layout tab you can click the Conditional button and then enter an expression that will control whether the text box is visible or not. You could make the text object visible based on the person using the report, the loaded data, time of day, or day of the week. Use it to alternate between two or more different text objects with different images. For example, in a corporate dashboard application you could make the image of a sunny day or of a stormy day appear based on the current month's revenue versus last month (or whatever passes for good news versus bad news in your business). The expression should be something that evaluates to a zero or a non-zero value. If the value is zero then the text object is not visible.