Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for building the library on Windows using MSVC. #56

Merged
merged 19 commits into from
Apr 16, 2024

Conversation

junhaoliao
Copy link
Member

@junhaoliao junhaoliao commented Mar 31, 2024

References

Internally, it was found installing clp-ffi-py on Windows 11 gave errors. It was determined that the following needs to be added to support installation on Windows:

  1. MSVC equivalent Extension.extra_compile_args in setup.py.
  2. MSVC compatible bswap implementations.
  3. MSVC compatible argument types to function std::from_chars in <clp-repo-root>/components/core/src/string_utils.inc:convert_string_to_int.

Interdependent PR

y-scope/clp#341

Description

  1. Requirement 2 & 3 are implemented core: Add support to compile clp::ffi component using MSVC. clp#341 .
  2. This PR implements Requirement 1 from above.
  3. This PR also adds support for running cibuildwheel on Windows (AMD64) in GitHub Actions.

Validation performed

  1. On a Windows computer, installed Python 3.11.1 (tags/v3.11.1:a7a450f, Dec 6 2022, 19:58:39) [MSC v.1934 64 bit (AMD64)] on win32.
  2. PowerShell: cd e:\; git clone --recursive -b windows-install https://github.com/junhaoliao/clp-ffi-py.git
  3. Initialized a Python repository with virtual environment set up.
  4. In the Python repository, pip install E:\clp-ffi-py.
  5. Observed installation success.
  6. Generated a random log file with below code:
    import math
    import random
    import time
    
    from clp_ffi_py.ir import FourByteEncoder
    import zstandard
    
    timestamp_ms: int = math.floor(time.time() * 1000)
    timestamp_format: str = "yy/MM/dd HH:mm:ss"
    timezone_id: str = "America/Toronto"
    
    cctx = zstandard.ZstdCompressor(level=22)
    with open('test.clp.zst', mode="wb") as fh:
        with cctx.stream_writer(fh) as compressor:
            compressor.write(FourByteEncoder.encode_preamble(timestamp_ms, timestamp_format, timezone_id))
            for i in range(100):
                compressor.write(FourByteEncoder.encode_message_and_timestamp_delta(random.randrange(0, 1000),
                                                                                    b"hello world\n"))
            compressor.write(FourByteEncoder.encode_end_of_ir())
  7. Viewed the generated ZStandard-compressed IR file on https://yscope.com/log-viewer, which loaded successfully.

.gitmodules Outdated
@@ -1,3 +1,4 @@
[submodule "src/clp"]
path = src/clp
url = https://github.com/y-scope/clp.git
url = https://github.com/junhaoliao/clp.git
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These shall not be merged

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR is merged with the latest CLP code: #55

Copy link
Member

@LinZhihao-723 LinZhihao-723 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments:

  1. Can u try to run the unit tests under the Windows dev environment?
  2. To properly build for Windows, we should also configure cibuildwheel to build the library for different Python versions. We can delay this to the next PR if u want, but can u try if we can execute cibuildwheel locally?

@@ -37,6 +37,9 @@
"src/clp_ffi_py/utils.cpp",
],
extra_compile_args=[
"/std:c++20",
"/O2,"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not O3?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

O3 isn't a valid option for MSVC.

MSVC's /O2, which "sets a combination of optimizations that optimizes code for maximum speed", is roughly the equivalent of GCC's -O3.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kk, good to know that, thx!

@junhaoliao
Copy link
Member Author

Some comments:

  1. Can u try to run the unit tests under the Windows dev environment?
  2. To properly build for Windows, we should also configure cibuildwheel to build the library for different Python versions. We can delay this to the next PR if u want, but can u try if we can execute cibuildwheel locally?
  1. All 41 unit tests passed in a 3.11 Virtual Environment on Windows.

    Test logs (.venv-3.11) PS E:\clp-ffi-py> python -m unittest -bv test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderDecompress.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderDecompressZstd.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderDecompressDefaultQuery.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderDecompressZstdDefaultQuery.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderTimeRangeQuery.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderTimeRangeQueryZstd.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_readers.TestCaseReaderTimeRangeQuery.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_readers.TestCaseReaderTimeRangeQueryZstd.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderWildcardQuery.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderWildcardQueryZstd.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_readers.TestCaseReaderWildcardQuery.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_readers.TestCaseReaderWildcardQueryZstd.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderTimeRangeWildcardQuery.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_decoder.TestCaseDecoderTimeRangeWildcardQueryZstd.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_readers.TestCaseReaderTimeRangeWildcardQuery.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_readers.TestCaseReaderTimeRangeWildcardQueryZstd.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_readers.TestCaseReaderDecompress.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_decoder_with_random_logs (test_ir.test_readers.TestCaseReaderDecompressZstd.test_decoder_with_random_logs) Tests encoding/decoding methods. ... ok test_buffer_protocol (test_ir.test_decoder_buffer.TestCaseDecoderBuffer.test_buffer_protocol) Tests whether the buffer protocol is disabled by default. ... ok test_streaming_default_buffer (test_ir.test_decoder_buffer.TestCaseDecoderBuffer.test_streaming_default_buffer) Tests DecoderBuffer's functionality using the default buffer capacity. ... ok test_streaming_large_buffer (test_ir.test_decoder_buffer.TestCaseDecoderBuffer.test_streaming_large_buffer) Tests DecoderBuffer's functionality using the large buffer capacity. ... ok test_streaming_small_buffer (test_ir.test_decoder_buffer.TestCaseDecoderBuffer.test_streaming_small_buffer) Tests DecoderBuffer's functionality using the small buffer capacity. ... ok test_encoding_methods_consistency (test_ir.test_encoder.TestCaseFourByteEncoder.test_encoding_methods_consistency) This test checks if the result of encode_message_and_timestamp_delta is ... ok test_init (test_ir.test_encoder.TestCaseFourByteEncoder.test_init) ... ok test_formatted_message (test_ir.test_log_event.TestCaseLogEvent.test_formatted_message) Test the reconstruction of the raw message. ... ok test_init (test_ir.test_log_event.TestCaseLogEvent.test_init) Test the initialization of LogEvent object without using keyword. ... ok test_keyword_init (test_ir.test_log_event.TestCaseLogEvent.test_keyword_init) Test the initialization of LogEvent object using keyword. ... ok test_pickle (test_ir.test_log_event.TestCaseLogEvent.test_pickle) Test the reconstruction of LogEvent object from pickling data. ... ok test_init (test_ir.test_metadata.TestCaseMetadata.test_init) Test the initialization of Metadata object without using keyword. ... ok test_keyword_init (test_ir.test_metadata.TestCaseMetadata.test_keyword_init) Test the initialization of Metadata object using keyword. ... ok test_timezone_new_reference (test_ir.test_metadata.TestCaseMetadata.test_timezone_new_reference) Test the timezone is a new reference returned. ... ok test_init (test_ir.test_query.TestCaseWildcardQuery.test_init) Test the initialization of WildcardQuery object. ... ok test_init_search_time (test_ir.test_query.TestCaseQuery.test_init_search_time) Test the construction of Query object with the different search time ... ok test_init_wildcard_queries (test_ir.test_query.TestCaseQuery.test_init_wildcard_queries) Test the construction of Query object with wildcard queries. ... ok test_log_event_match (test_ir.test_query.TestCaseQuery.test_log_event_match) Test the match between a Query object and a LogEvent object. ... ok test_pickle (test_ir.test_query.TestCaseQuery.test_pickle) Test the reconstruction of Query object from pickling data. ... ok test_exception (test_ir.test_query_builder.TestCaseQueryBuilder.test_exception) Tests whether QueryBuilderException is triggered as expected. ... ok test_init (test_ir.test_query_builder.TestCaseQueryBuilder.test_init) Tests the default initialized Query Builder and its behavior. ... ok test_set_value (test_ir.test_query_builder.TestCaseQueryBuilder.test_set_value) Tests QueryBuilder by building different Query objects. ... ok test_allow_incomplete_ir_stream_error (test_ir.test_readers.TestIncompleteIRStream.test_allow_incomplete_ir_stream_error) Tests the reader against an incomplete IR file with ... ok test_incomplete_ir_stream_error (test_ir.test_readers.TestIncompleteIRStream.test_incomplete_ir_stream_error) Tests the reader against an incomplete IR file with ... ok

    Ran 41 tests in 0.768s

    OK

@junhaoliao junhaoliao changed the title [WIP] setup: Add support for installing the lib on Windows with MSVC. setup: Add support for installing the lib on Windows with MSVC. Apr 1, 2024
@junhaoliao
Copy link
Member Author

Some comments:

  1. Can u try to run the unit tests under the Windows dev environment?
  2. To properly build for Windows, we should also configure cibuildwheel to build the library for different Python versions. We can delay this to the next PR if u want, but can u try if we can execute cibuildwheel locally?
  1. The GitHub Action workflow is now modified.

pyproject.toml Outdated
[tool.cibuildwheel.windows]
archs = ["AMD64"]
test-command = [
"cd /d {package}\\tests",
Copy link
Member Author

@junhaoliao junhaoliao Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The /d option must be specified or this error message gets reported: The system cannot find the path specified. That is because the project is synced into D:\ drive in GitHub Action workflows, which is a non-system (non-C:\) drive.

Reference: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/cd#:~:text=Description-,/d,-Changes%20the%20current

Copy link
Member Author

@junhaoliao junhaoliao Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the \ needs to be escaped. / might also work though.

Copy link
Member

@kirkrodrigues kirkrodrigues Apr 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you test if "python -m unittest discover -s {package}/tests -fv" works on its own (rather than cd + unittest)?

@junhaoliao junhaoliao requested a review from LinZhihao-723 April 1, 2024 02:43
README.md Outdated
@@ -19,7 +19,7 @@ python3 -m pip install --upgrade clp-ffi-py
Note:

- Python 3.7 or higher is required.
- Only Linux and macOS are supported at present.
- Tested only on Linux, macOS and Windows.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need work "only", lol

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol i was thinking about Solaris and FreeBSD when i wrote this, but yeah the word seems a bit unnecessary here

@@ -37,6 +37,9 @@
"src/clp_ffi_py/utils.cpp",
],
extra_compile_args=[
"/std:c++20",
"/O2,"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kk, good to know that, thx!

.gitmodules Outdated
@@ -1,3 +1,4 @@
[submodule "src/clp"]
path = src/clp
url = https://github.com/y-scope/clp.git
url = https://github.com/junhaoliao/clp.git
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR is merged with the latest CLP code: #55

LinZhihao-723
LinZhihao-723 previously approved these changes Apr 15, 2024
Copy link
Member

@LinZhihao-723 LinZhihao-723 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final comments

README.md Outdated Show resolved Hide resolved
Copy link
Member

@LinZhihao-723 LinZhihao-723 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about the following commit message:
Add support for building the library on Windows using MSVC. (#56)

@junhaoliao junhaoliao changed the title setup: Add support for installing the lib on Windows with MSVC. Add support for building the library on Windows using MSVC. Apr 16, 2024
@junhaoliao junhaoliao merged commit d52c8b7 into y-scope:main Apr 16, 2024
60 checks passed
@junhaoliao junhaoliao deleted the windows-install branch April 16, 2024 21:42
@junhaoliao
Copy link
Member Author

Can I help release a new version on PyPI? I assume we also need to update clp-loglib-py, right?

@LinZhihao-723
Copy link
Member

Can I help release a new version on PyPI? I assume we also need to update clp-loglib-py, right?

Sure, I will do it shortly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants