Replace compressor_frontend with log-surgeon submodule. #131

SharafMohamed · 2023-06-05T05:01:21Z

Reference

Depends on PR to: y-scope/log-surgeon#1

Description

Replaced compressor_frontend with log_surgeon

Removed compressor_frontend folder
Removed compressor_frontend from CMakeLists.txt
Added log_surgeon to CMakeLists.txt and cmake/utils.cmake
Changed includes from "compressor_frontend/" to <log_surgeon/>
Replace compressor_frontend::* with log_surgeon::* counterparts
Moved load_lexer_from_file from compressor_frontend to Utils.cpp
fixed test-ParserWithUserSchema.cpp to match new throw messages from log_surgeon

Moved QueryToken class into its own files

Removed from Grep.cpp and Grep.hpp
Moved to QueryToken.cpp and QueryToken.hpp
Added to CMakeLists.txt
Seperated query functionality for heuristic and schema in Grep.cpp

Created SearchToken class

wraps log_surgeon::Token
contain std::set m_type_ids_set

Renamed parse_and_encode to parse_and_encode_with_library

Validation performed

Ran on clp-benchmark (Loghub, Hadoop, Openstack, Spark, Var-logs)

Compressed succesfully
Decompressed logs match original

Ran with all executable and unit tests in CLP

- Removed compressor_frontend from cmakelists - Added log_surgeon to cmakelists

- Unit tests all work

components/core/cmake/utils.cmake

components/core/src/Grep.hpp

components/core/tests/test-Stopwatch.cpp

…ightly longer than expected

components/core/CMakeLists.txt

# Conflicts: # components/core/CMakeLists.txt # components/core/src/Utils.cpp # components/core/src/compressor_frontend/Lexer.tpp # components/core/src/compressor_frontend/LogParser.cpp # components/core/src/compressor_frontend/SchemaParser.cpp # components/core/src/compressor_frontend/finite_automata/RegexAST.tpp # components/core/src/streaming_archive/writer/Archive.cpp # components/core/tests/test-Stopwatch.cpp

- Fixed unit-test in CMakeLists to include log_surgeon

# Conflicts: # components/core/src/streaming_archive/writer/Archive.cpp # components/core/src/streaming_archive/writer/Archive.hpp

components/core/src/Grep.cpp

components/core/src/QueryToken.cpp

components/core/src/Utils.cpp

-Switch const auto& to be auto const&

… causing the heuristic to not store variable segment indicies correctly

… with schema; Ideally should use a set, but its not currently initialized

…nor refactoring.

kirkrodrigues

We also need to fix the workflows so that they use g++-10.

components/core/src/clp/FileCompressor.cpp

components/core/src/ReaderInterface.hpp

components/core/src/Grep.cpp

components/core/src/streaming_archive/writer/Archive.hpp

…old_ts_pattern related code

SharafMohamed added 3 commits May 31, 2023 18:13

- Removed compressor_frontend folder

2fc9cf5

- Removed compressor_frontend from cmakelists - Added log_surgeon to cmakelists

Merge remote-tracking branch 'upstream/main' into main

b30e9c5

- Everything builds with log_surgeon

bebcf98

- Unit tests all work

SharafMohamed force-pushed the main branch from d78922d to bebcf98 Compare June 26, 2023 20:49

davidlion reviewed Jun 27, 2023

View reviewed changes

components/core/cmake/utils.cmake Outdated Show resolved Hide resolved

components/core/src/Grep.hpp Outdated Show resolved Hide resolved

components/core/src/Grep.hpp Outdated Show resolved Hide resolved

components/core/tests/test-Stopwatch.cpp Outdated Show resolved Hide resolved

SharafMohamed added 2 commits July 21, 2023 15:27

Removed whitespace at end of lines

1af7e69

Removed multiple measurement test that keeps failing due to taking sl…

707ff06

…ightly longer than expected

davidlion reviewed Jul 23, 2023

View reviewed changes

components/core/CMakeLists.txt Outdated Show resolved Hide resolved

SharafMohamed added 9 commits July 24, 2023 03:52

added log_surgeon as submodule

395345a

Updated includes for log-surgeon

165919c

Fixed missing changes to log-surgeon includes

12efe93

- Changed log_surgeon and yaml-cpp includes to be cleaner

c90d009

- Fixed unit-test in CMakeLists to include log_surgeon

added log_surgeon to third-party regex in clange-format

e47a144

Fixed comments

40c92fa

Merge remote-tracking branch 'upstream/main' into main

2b14e7f

# Conflicts: # components/core/src/streaming_archive/writer/Archive.cpp # components/core/src/streaming_archive/writer/Archive.hpp

Added space to comment

c595474

davidlion requested changes Jul 25, 2023

View reviewed changes

SharafMohamed added 6 commits July 26, 2023 17:07

Updated log-surgeon submodule to be at the correct commit

e33da29

Cleaned up grep.cpp

78bec44

Cleaned up Grep.hpp

51f0494

Cleaned up QueryToken cpp and hpp

5d79a0b

Cleaned up clg.cpp

8ba0491

-Fixed ordering in CMakeLists

6a86479

-Switch const auto& to be auto const&

davidlion mentioned this pull request Jul 28, 2023

Add build workflows; fix macos build errors/warnings; update code formatting y-scope/log-surgeon#5

Merged

SharafMohamed added 5 commits July 31, 2023 11:19

Cleaned up FileCompressor.cpp

e42e275

Cleaned up FileCompressor.hpp

b522e60

Cleaned up compression.hpp

7bc4304

Updated doc string in compression.hpp

a5c4336

Cleaned up test-Grep.cpp

8f5b291

Merge remote-tracking branch 'upstream/main' into main

f21b77f

SharafMohamed mentioned this pull request Sep 18, 2023

First token in non-timestamped logs now correctly handled; First character of each line in non-timestamped logs now correctly handled y-scope/log-surgeon#9

Merged

SharafMohamed and others added 9 commits September 25, 2023 11:15

Pulled latest version of log-surgeon

fa4dd3f

Merge remote-tracking branch 'upstream/main' into main

0e4a6b4

Fixed update_segment_indices to use the passed in parameter, this was…

d8ffc74

… causing the heuristic to not store variable segment indicies correctly

Removed some redundancies in grep

e3e6911

Correctly use the type vector when checking search_token type in grep…

120342a

… with schema; Ideally should use a set, but its not currently initialized

Merge remote-tracking branch 'upstream/main' into main

8ec7713

Undo changes unrelated to PR; Minor refactoring.

14cadd2

Undo changes unrelated to PR (move QueryToken back into Grep.cpp): Mi…

04888cf

…nor refactoring.

Move SearchToken into Grep.cpp to limit scope.

76cf701

kirkrodrigues requested changes Nov 17, 2023

View reviewed changes

Switch to gcc-10

a2578ff

SharafMohamed mentioned this pull request Nov 20, 2023

Add LogEventView as a member within LogParser y-scope/log-surgeon#11

Merged

SharafMohamed and others added 8 commits November 20, 2023 20:29

Fixed schema to properly segment non-timestamped files; Cleaned up m_…

96772cb

…old_ts_pattern related code

Merge branch 'main' of https://github.com/SharafMohamed/clp into main

1b07fdf

Move LogSurgeonReader into its own file

bd2ff45

Removed static declaration of LogEventView

1ae4eb4

Fixed CLG CLO and unit-tests to compile with new changes

fc65297

Undo some unnecessary changes; Minor refactoring.

772fc56

Merge remote-tracking branch 'upstream/main' into main

d56af97

Merge remote-tracking branch 'upstream/main' into main

86c0aa1

kirkrodrigues mentioned this pull request Dec 5, 2023

Drop support for building core on Ubuntu Bionic and add support for building it on Jammy. #176

Merged

kirkrodrigues approved these changes Dec 6, 2023

View reviewed changes

Undo unrelated formatting changes.

c917350

kirkrodrigues merged commit 6571d8d into y-scope:main Dec 6, 2023
4 checks passed

kirkrodrigues mentioned this pull request Dec 13, 2023

Add log-surgeon to dependencies downloadable without git. #180

Merged

kirkrodrigues mentioned this pull request Dec 28, 2023

Add log-surgeon source-dependency to README. #197

Merged

kirkrodrigues mentioned this pull request Apr 29, 2024

Refactor Ubuntu Focal gcc-10 setup and update related usage instructions. #371

Merged

kirkrodrigues changed the title ~~Log Parsing Library~~ Replace compressor_frontend with log-surgeon submodule. May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace compressor_frontend with log-surgeon submodule. #131

Replace compressor_frontend with log-surgeon submodule. #131

SharafMohamed commented Jun 5, 2023 •

edited

Loading

kirkrodrigues left a comment

Replace compressor_frontend with log-surgeon submodule. #131

Replace compressor_frontend with log-surgeon submodule. #131

Conversation

SharafMohamed commented Jun 5, 2023 • edited Loading

Reference

Description

Validation performed

kirkrodrigues left a comment

Choose a reason for hiding this comment

SharafMohamed commented Jun 5, 2023 •

edited

Loading