Reviewing the Execution Logs of Katalon Studio

While the output logs of Katalon Studio are extremely helpful, it’s not possible (as far as I can tell) to copy that output directly. The reason being, I want to run a baseline test to capture multiple data points. After a code deployment, I want to run the tests again and verify the results are the same. The logs look great in Katalon, but the raw logs contain a lot more information.

On my Mac, I can use Terminal and the Unix command, Grep, to start filtering. The cleanest file to work with is the JUnit_Report.xml file, located in the Reports directory. Inside, there are lines marked as:

[MESSAGE][PASSED]
[MESSAGE][WARNING]
[MESSAGE][FAILED]
[MESSAGE][ERROR]

The real file will contain lines that look like this:

2018-03-25 09:42:40 - [TEST_STEP][PASSED] - Statement - today = new java.util.Date(): null
2018-03-25 09:42:40 - [TEST_STEP][PASSED] - Statement - yesterday = today.previous(): null
2018-03-25 09:42:40 - [TEST_STEP][PASSED] - Statement - todayDate = today.format("MM/dd/yyyy"): null
2018-03-25 10:02:27 - [MESSAGE][PASSED] - Delayed 2 second(s)
2018-03-25 10:02:27 - [MESSAGE][WARNING] - Filter Results: 1679
2018-03-25 10:02:27 - [MESSAGE][PASSED] - Text of object 'Object Repository/Page_/Dashboard/Footer-Total Number of Users Returned' is: '1-25 of 1679 users'
2018-03-25 10:02:27 - [MESSAGE][WARNING] - Pagination Results: 1679
2018-03-25 10:02:27 - [MESSAGE][PASSED] - SUCCESS: The Filter Results Matches the Pagination Results]]></system-out>

The drawback are the [TEST_STEP][PASSED] lines. Those need to be filtered out.

At the Terminal prompt:

grep "MESSAGE" JUnit_Report.xml > firstpass.txt

This creates a text file that contains the [MESSAGE] lines.

This is good, but there will be hundreds of [MESSAGE][PASSED] lines, which really aren’t that important. That’s not a problem as those lines can be filtered out in LibreOffice using the AutoFilter.

Load the file, select AutoFilter and the option for Standard Filter. Set the filter so the lines that Do not contain the word PASSED are displayed.

Now we have our Warning, Error and Failed messages together. When the second test is completed and the results filtered in the same manner, they can be pasted into the next column to be compared line by line. Or, the EXACT function can be run to compare the two strings.

To illustrate, here is the Log Viewer from Katalon:

And here are the same results in LibreOffice:

If needed, the same filtering method can be used to see the Failed and Error messages. But this gives the ability to run a test, capture data, run the test again and compare the results.