ULP (Universal Log Parsing) is a highly accurate log parsing tool, the ability to extract templates from unstructured log data. ULP learns from sample log data to recognize future log events. It combines pattern matching and frequency analysis techniques. First, log events are organized into groups using a text processing method. Frequency analysis is then applied locally to instances of the same group to identify static and dynamic content of log events. When applied to 10 log datasets of the Loghub benchmark, ULP achieves an average accuracy of 89.2%, which outperforms the accuracy of four leading log parsing tools, namely Drain, Logram, Spell and AEL. Additionally, ULP can parse up to four million log events in less than 3 minutes. ULP can be readily used by practitioners and researchers to parse effectively and efficiently large log files so as to support log analysis tasks.

The code has been tested in the following enviornment:

• python 3.7.6
• regex 2022.3.2
• pandas 1.0.1
• numpy 1.18.1
• scipy 1.4.1

Run the following scripts to start the demo:

python demo.py

Run the following scripts to execute the benchmark:

python benchmark.py

Running the benchmark script on Loghub_2k datasets, you could obtain the following results.

If you use the code or benchmarking results in your publication, please kindly cite the following papers.

• [ICSME'22] Issam Sedki, Abdelwahab Hamou-Lhadj, Otmane Ait-Mohamed, Mohammed A. Shehab. An Effective Approach for Parsing Large Log Files, Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), 2022.
• [ICSE'19] Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, Michael R. Lyu. Tools and Benchmarks for Automated Log Parsing. International Conference on Software Engineering (ICSE), 2019.