Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace compressor_frontend with log-surgeon submodule. #131

Merged
merged 65 commits into from
Dec 6, 2023
+658 −4,917
Merged

Replace compressor_frontend with log-surgeon submodule. #131

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
2fc9cf5
- Removed compressor_frontend folder
SharafMohamed May 31, 2023
b30e9c5
Merge remote-tracking branch 'upstream/main' into main
SharafMohamed May 31, 2023
bebcf98
- Everything builds with log_surgeon
SharafMohamed Jun 1, 2023
1af7e69
Removed whitespace at end of lines
SharafMohamed Jul 21, 2023
707ff06
Removed multiple measurement test that keeps failing due to taking sl…
SharafMohamed Jul 21, 2023
395345a
added log_surgeon as submodule
SharafMohamed Jul 24, 2023
e064716
Merge remote-tracking branch 'upstream/main' into main
SharafMohamed Jul 24, 2023
165919c
Updated includes for log-surgeon
SharafMohamed Jul 24, 2023
12efe93
Fixed missing changes to log-surgeon includes
SharafMohamed Jul 24, 2023
c90d009
- Changed log_surgeon and yaml-cpp includes to be cleaner
SharafMohamed Jul 24, 2023
e47a144
added log_surgeon to third-party regex in clange-format
SharafMohamed Jul 24, 2023
40c92fa
Fixed comments
SharafMohamed Jul 24, 2023
2b14e7f
Merge remote-tracking branch 'upstream/main' into main
SharafMohamed Jul 24, 2023
c595474
Added space to comment
SharafMohamed Jul 24, 2023
e33da29
Updated log-surgeon submodule to be at the correct commit
SharafMohamed Jul 26, 2023
78bec44
Cleaned up grep.cpp
SharafMohamed Jul 26, 2023
51f0494
Cleaned up Grep.hpp
SharafMohamed Jul 26, 2023
5d79a0b
Cleaned up QueryToken cpp and hpp
SharafMohamed Jul 26, 2023
8ba0491
Cleaned up clg.cpp
SharafMohamed Jul 26, 2023
6a86479
-Fixed ordering in CMakeLists
SharafMohamed Jul 27, 2023
e42e275
Cleaned up FileCompressor.cpp
SharafMohamed Jul 31, 2023
b522e60
Cleaned up FileCompressor.hpp
SharafMohamed Jul 31, 2023
7bc4304
Cleaned up compression.hpp
SharafMohamed Jul 31, 2023
a5c4336
Updated doc string in compression.hpp
SharafMohamed Jul 31, 2023
8f5b291
Cleaned up test-Grep.cpp
SharafMohamed Jul 31, 2023
bd21621
Cleaned up test-ParserWithUserSchema.cpp
SharafMohamed Jul 31, 2023
11d76f3
Cleaned up Archive.cpp
SharafMohamed Jul 31, 2023
661b2e9
Fixed doc string and cleaned up Archive.hpp
SharafMohamed Jul 31, 2023
ae2f63f
Cleaned up Utils.cpp
SharafMohamed Jul 31, 2023
a689eb0
Better documented TODOs: mainly about removing duplicated code by add…
SharafMohamed Aug 1, 2023
8b395a8
Updated TODO; Now using try_schema_file when possible
SharafMohamed Aug 7, 2023
27aeb2b
Updated TODO
SharafMohamed Aug 7, 2023
a008882
Turned reader_wrapper from a lambda into a class inheriting from log_…
SharafMohamed Aug 7, 2023
889f2f7
updated log_surgeon submodule
SharafMohamed Aug 7, 2023
8e6594f
Fixed naming for StringReader and FileReader shared_ptrs
SharafMohamed Aug 9, 2023
d4f28ce
Made shared_ptr to Reader a reference in ReaderInterfaceWrapper
SharafMohamed Aug 9, 2023
96e5df2
Fixed ReaderInterfaceWrapper to correctly set Reader::read that was p…
SharafMohamed Aug 16, 2023
fee6fd4
Removed unneeded pos_processed_string var in get_bounds_of_next_poten…
SharafMohamed Aug 16, 2023
ed23d9e
Removed post_processed_search_string in Grep.cpp
SharafMohamed Aug 16, 2023
e6315ec
Updated to match the allowance of multiple delimiters lines in log_su…
SharafMohamed Aug 25, 2023
66cdf5c
Updated log-surgeon to the newest commit.
SharafMohamed Sep 11, 2023
23f7b61
Updated example log to have floats
SharafMohamed Sep 11, 2023
0861ce3
Merge remote-tracking branch 'upstream/main' into main
SharafMohamed Sep 13, 2023
a271e0c
Fixed double to float
SharafMohamed Sep 17, 2023
7386f5a
Fixed bug where first char of first token would become static text ev…
SharafMohamed Sep 17, 2023
f21b77f
Merge remote-tracking branch 'upstream/main' into main
SharafMohamed Sep 17, 2023
fa4dd3f
Pulled latest version of log-surgeon
SharafMohamed Sep 25, 2023
0e4a6b4
Merge remote-tracking branch 'upstream/main' into main
SharafMohamed Sep 29, 2023
d8ffc74
Fixed update_segment_indices to use the passed in parameter, this was…
SharafMohamed Oct 2, 2023
e3e6911
Removed some redundancies in grep
SharafMohamed Oct 2, 2023
120342a
Correctly use the type vector when checking search_token type in grep…
SharafMohamed Oct 2, 2023
8ec7713
Merge remote-tracking branch 'upstream/main' into main
SharafMohamed Nov 13, 2023
14cadd2
Undo changes unrelated to PR; Minor refactoring.
kirkrodrigues Nov 17, 2023
04888cf
Undo changes unrelated to PR (move QueryToken back into Grep.cpp): Mi…
kirkrodrigues Nov 17, 2023
76cf701
Move SearchToken into Grep.cpp to limit scope.
kirkrodrigues Nov 17, 2023
a2578ff
Switch to gcc-10
kirkrodrigues Nov 18, 2023
96772cb
Fixed schema to properly segment non-timestamped files; Cleaned up m_…
SharafMohamed Nov 21, 2023
1b07fdf
Merge branch 'main' of https://github.com/SharafMohamed/clp into main
SharafMohamed Nov 21, 2023
bd2ff45
Move LogSurgeonReader into its own file
SharafMohamed Nov 21, 2023
1ae4eb4
Removed static declaration of LogEventView
SharafMohamed Nov 22, 2023
fc65297
Fixed CLG CLO and unit-tests to compile with new changes
SharafMohamed Nov 22, 2023
772fc56
Undo some unnecessary changes; Minor refactoring.
kirkrodrigues Nov 24, 2023
d56af97
Merge remote-tracking branch 'upstream/main' into main
SharafMohamed Nov 27, 2023
86c0aa1
Merge remote-tracking branch 'upstream/main' into main
SharafMohamed Dec 5, 2023
c917350
Undo unrelated formatting changes.
kirkrodrigues Dec 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,6 @@
[submodule "components/core/submodules/yaml-cpp"]
path = components/core/submodules/yaml-cpp
url = https://github.com/jbeder/yaml-cpp.git
[submodule "components/core/submodules/log-surgeon"]
path = components/core/submodules/log-surgeon
url = https://github.com/y-scope/log-surgeon.git
98 changes: 14 additions & 84 deletions components/core/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ else()
endif()
message(STATUS "Building using ${CLP_LIBS_STRING} libraries")

# Add log surgeon
add_subdirectory(submodules/log-surgeon EXCLUDE_FROM_ALL)

# Link against c++fs if required by the compiler being used
set(STD_FS_LIBS "")
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
Expand Down Expand Up @@ -187,28 +190,6 @@ set(SOURCE_FILES_clp
src/clp/StructuredFileToCompress.hpp
src/clp/utils.cpp
src/clp/utils.hpp
src/compressor_frontend/Constants.hpp
src/compressor_frontend/finite_automata/RegexAST.hpp
src/compressor_frontend/finite_automata/RegexAST.tpp
src/compressor_frontend/finite_automata/RegexDFA.hpp
src/compressor_frontend/finite_automata/RegexDFA.tpp
src/compressor_frontend/finite_automata/RegexNFA.hpp
src/compressor_frontend/finite_automata/RegexNFA.tpp
src/compressor_frontend/finite_automata/UnicodeIntervalTree.hpp
src/compressor_frontend/finite_automata/UnicodeIntervalTree.tpp
src/compressor_frontend/LALR1Parser.cpp
src/compressor_frontend/LALR1Parser.hpp
src/compressor_frontend/LALR1Parser.tpp
src/compressor_frontend/Lexer.hpp
src/compressor_frontend/Lexer.tpp
src/compressor_frontend/LogParser.cpp
src/compressor_frontend/LogParser.hpp
src/compressor_frontend/SchemaParser.cpp
src/compressor_frontend/SchemaParser.hpp
src/compressor_frontend/Token.cpp
src/compressor_frontend/Token.hpp
src/compressor_frontend/utils.cpp
src/compressor_frontend/utils.hpp
src/database_utils.cpp
src/database_utils.hpp
src/Defs.h
Expand Down Expand Up @@ -262,6 +243,8 @@ set(SOURCE_FILES_clp
src/Profiler.hpp
src/Query.cpp
src/Query.hpp
src/QueryToken.cpp
src/QueryToken.hpp
src/ReaderInterface.cpp
src/ReaderInterface.hpp
src/spdlog_with_specializations.hpp
Expand Down Expand Up @@ -334,6 +317,7 @@ target_link_libraries(clp
PRIVATE
Boost::filesystem Boost::iostreams Boost::program_options
fmt::fmt
log_surgeon::log_surgeon
spdlog::spdlog
${sqlite_LIBRARY_DEPENDENCIES}
LibArchive::LibArchive
Expand All @@ -356,26 +340,6 @@ set(SOURCE_FILES_clg
src/clg/clg.cpp
src/clg/CommandLineArguments.cpp
src/clg/CommandLineArguments.hpp
src/compressor_frontend/Constants.hpp
src/compressor_frontend/finite_automata/RegexAST.hpp
src/compressor_frontend/finite_automata/RegexAST.tpp
src/compressor_frontend/finite_automata/RegexDFA.hpp
src/compressor_frontend/finite_automata/RegexDFA.tpp
src/compressor_frontend/finite_automata/RegexNFA.hpp
src/compressor_frontend/finite_automata/RegexNFA.tpp
src/compressor_frontend/finite_automata/UnicodeIntervalTree.hpp
src/compressor_frontend/finite_automata/UnicodeIntervalTree.tpp
src/compressor_frontend/LALR1Parser.cpp
src/compressor_frontend/LALR1Parser.hpp
src/compressor_frontend/LALR1Parser.tpp
src/compressor_frontend/Lexer.hpp
src/compressor_frontend/Lexer.tpp
src/compressor_frontend/SchemaParser.cpp
src/compressor_frontend/SchemaParser.hpp
src/compressor_frontend/Token.cpp
src/compressor_frontend/Token.hpp
src/compressor_frontend/utils.cpp
src/compressor_frontend/utils.hpp
src/database_utils.cpp
src/database_utils.hpp
src/Defs.h
Expand Down Expand Up @@ -421,6 +385,8 @@ set(SOURCE_FILES_clg
src/Profiler.hpp
src/Query.cpp
src/Query.hpp
src/QueryToken.cpp
src/QueryToken.hpp
src/ReaderInterface.cpp
src/ReaderInterface.hpp
src/spdlog_with_specializations.hpp
Expand Down Expand Up @@ -491,6 +457,7 @@ target_link_libraries(clg
PRIVATE
Boost::filesystem Boost::iostreams Boost::program_options
fmt::fmt
log_surgeon::log_surgeon
KQL
MariaDBClient::MariaDBClient
spdlog::spdlog
Expand All @@ -509,26 +476,6 @@ set(SOURCE_FILES_clo
src/clo/CommandLineArguments.hpp
src/clo/ControllerMonitoringThread.cpp
src/clo/ControllerMonitoringThread.hpp
src/compressor_frontend/Constants.hpp
src/compressor_frontend/finite_automata/RegexAST.hpp
src/compressor_frontend/finite_automata/RegexAST.tpp
src/compressor_frontend/finite_automata/RegexDFA.hpp
src/compressor_frontend/finite_automata/RegexDFA.tpp
src/compressor_frontend/finite_automata/RegexNFA.hpp
src/compressor_frontend/finite_automata/RegexNFA.tpp
src/compressor_frontend/finite_automata/UnicodeIntervalTree.hpp
src/compressor_frontend/finite_automata/UnicodeIntervalTree.tpp
src/compressor_frontend/LALR1Parser.cpp
src/compressor_frontend/LALR1Parser.hpp
src/compressor_frontend/LALR1Parser.tpp
src/compressor_frontend/Lexer.hpp
src/compressor_frontend/Lexer.tpp
src/compressor_frontend/SchemaParser.cpp
src/compressor_frontend/SchemaParser.hpp
src/compressor_frontend/Token.cpp
src/compressor_frontend/Token.hpp
src/compressor_frontend/utils.cpp
src/compressor_frontend/utils.hpp
src/database_utils.cpp
src/database_utils.hpp
src/Defs.h
Expand Down Expand Up @@ -564,6 +511,8 @@ set(SOURCE_FILES_clo
src/Profiler.hpp
src/Query.cpp
src/Query.hpp
src/QueryToken.cpp
src/QueryToken.hpp
src/ReaderInterface.cpp
src/ReaderInterface.hpp
src/spdlog_with_specializations.hpp
Expand Down Expand Up @@ -636,6 +585,7 @@ target_link_libraries(clo
PRIVATE
Boost::filesystem Boost::iostreams Boost::program_options
fmt::fmt
log_surgeon::log_surgeon
msgpack-cxx
spdlog::spdlog
${sqlite_LIBRARY_DEPENDENCIES}
Expand Down Expand Up @@ -665,28 +615,6 @@ set(SOURCE_FILES_unitTest
src/clp/StructuredFileToCompress.hpp
src/clp/utils.cpp
src/clp/utils.hpp
src/compressor_frontend/Constants.hpp
src/compressor_frontend/finite_automata/RegexAST.hpp
src/compressor_frontend/finite_automata/RegexAST.tpp
src/compressor_frontend/finite_automata/RegexDFA.hpp
src/compressor_frontend/finite_automata/RegexDFA.tpp
src/compressor_frontend/finite_automata/RegexNFA.hpp
src/compressor_frontend/finite_automata/RegexNFA.tpp
src/compressor_frontend/finite_automata/UnicodeIntervalTree.hpp
src/compressor_frontend/finite_automata/UnicodeIntervalTree.tpp
src/compressor_frontend/LALR1Parser.cpp
src/compressor_frontend/LALR1Parser.hpp
src/compressor_frontend/LALR1Parser.tpp
src/compressor_frontend/Lexer.hpp
src/compressor_frontend/Lexer.tpp
src/compressor_frontend/LogParser.cpp
src/compressor_frontend/LogParser.hpp
src/compressor_frontend/SchemaParser.cpp
src/compressor_frontend/SchemaParser.hpp
src/compressor_frontend/Token.cpp
src/compressor_frontend/Token.hpp
src/compressor_frontend/utils.cpp
src/compressor_frontend/utils.hpp
src/database_utils.cpp
src/database_utils.hpp
src/Defs.h
Expand Down Expand Up @@ -765,6 +693,8 @@ set(SOURCE_FILES_unitTest
src/Profiler.hpp
src/Query.cpp
src/Query.hpp
src/QueryToken.cpp
src/QueryToken.hpp
src/ReaderInterface.cpp
src/ReaderInterface.hpp
src/spdlog_with_specializations.hpp
Expand Down
5 changes: 5 additions & 0 deletions components/core/cmake/utils.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,14 @@ set(SOURCE_FILES_make-dictionaries-readable
${CMAKE_CURRENT_SOURCE_DIR}/submodules/date/include/date/date.h
)
add_executable(make-dictionaries-readable ${SOURCE_FILES_make-dictionaries-readable})
target_include_directories(make-dictionaries-readable
PRIVATE
${CMAKE_SOURCE_DIR}/submodules
)
target_link_libraries(make-dictionaries-readable
PRIVATE
Boost::filesystem Boost::iostreams Boost::program_options
log_surgeon::log_surgeon
spdlog::spdlog
ZStd::ZStd
)
Expand Down
2 changes: 1 addition & 1 deletion components/core/src/FileReader.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ void FileReader::open (const string& path) {
ErrorCode error_code = try_open(path);
if (ErrorCode_Success != error_code) {
if (ErrorCode_FileNotFound == error_code) {
throw "File not found: " + boost::filesystem::weakly_canonical(path).string() + "\n";
throw "File not found: " + boost::filesystem::weakly_canonical(path).string() + "\n";
} else {
throw OperationFailed(error_code, __FILENAME__, __LINE__);
}
Expand Down
Loading