Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pre-processing options work for PreTrainedVectorizers [resolves #307] #308

Open
wants to merge 15 commits into
base: develop
Choose a base branch
from
Open

Pre-processing options work for PreTrainedVectorizers [resolves #307] #308

wants to merge 15 commits into from

Conversation

ertugrul-dmr
Copy link
Contributor

  • Added proper ways to use pre-processing in transformer based vectorizers where they used to work on raw strings.
  • Added ability to use pre-trained vectorizers and Text2Doc together in pipelines.
  • Text2Doc settings should be properly changing after altering the settings.
  • Added test cases for changes mentioned above.

dafajon and others added 15 commits December 22, 2021 16:26
@dafajon
Implement docstrings for Document object.
cb90126
@ertugrul-dmr
Initial docstrings for extension an prebuilt files
60c38dc
@dafajon
Add raises to relevant docstrings.
b7a6d3e
@dafajon
Implement docstrings for sentences.
d8aa70c
@dafajon
Add docstrings for BM25 Impl
501616f
@dafajon
Implement docstring for TFImpl
117e819
@dafajon
Implement docstring for Span class
3867eae
@dafajon
Implement docstring for DocBuilder
4317e8f
@ertugrul-dmr
Added method desc for sklearn.py classes
2551444
@ertugrul-dmr
Initial addition for fixing the case where doc settings not applying …
2712f5e 
…to pretrained vectorizers.
@dafajon
Remove Doc class attribute from vectorizer.
ce45f2e
@dafajon
Implement existing Doc attribute check to Text2Doc initalization to e…
1a22037 
…nsure changes in arguments take effect.
@dafajon
Fix tests for new implementation.
6ecc985
@dafajon
Implement test for text2doc setting change in the runtime.
0225827
@ertugrul-dmr
Added preprocess fix for bert_embeddings.
e3a765b
@ertugrul-dmr ertugrul-dmr added the enhancement New feature or request label Jan 5, 2022
@ertugrul-dmr ertugrul-dmr requested a review from dafajon January 5, 2022 11:52
@ertugrul-dmr ertugrul-dmr changed the title Pre-processing options work for PreTrainedVectorizer[resolves #307] Pre-processing options work for PreTrainedVectorizer [resolves #307] Jan 5, 2022
@ertugrul-dmr ertugrul-dmr linked an issue Jan 5, 2022 that may be closed by this pull request
Make Pre-processing options work for PreTrainedVectorizer #307
Open
@ertugrul-dmr ertugrul-dmr changed the title Pre-processing options work for PreTrainedVectorizer [resolves #307] Pre-processing options work for PreTrainedVectorizers [resolves #307] Jan 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dafajon dafajon Awaiting requested review from dafajon

Assignees

@dafajon dafajon

@ertugrul-dmr ertugrul-dmr

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Make Pre-processing options work for PreTrainedVectorizer
2 participants
@ertugrul-dmr @dafajon