Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support single input file #67

Merged
merged 8 commits into from
Nov 4, 2023
Merged

Support single input file #67

merged 8 commits into from
Nov 4, 2023

Conversation

awdeorio
Copy link
Contributor

@awdeorio awdeorio commented Nov 3, 2023

Add support for a single input file.

Validation

$ tree crawl/
crawl/
├── 00001.html
├── 00002.html
├── 00003.html
└── MANIFEST.txt

1 directory, 4 files

$ cat crawl/MANIFEST.txt 
00001.html
00002.html
00003.html

$ rm -rf output
madoop \
  -input crawl/MANIFEST.txt \
  -output output \
  -mapper /bin/cat \
  -reducer /bin/cat
cat output/*
INFO: Starting map stage
INFO: Finished map executions: 1
INFO: Starting group stage
INFO: Starting reduce stage
INFO: Finished reduce executions: 3
INFO: Output directory: output
00001.html
00002.html
00003.html

Copy link

codecov bot commented Nov 3, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Files Coverage Δ
madoop/__main__.py 93.75% <ø> (ø)
madoop/mapreduce.py 96.96% <100.00%> (+0.19%) ⬆️

📢 Thoughts on this report? Let us know!.

Copy link
Contributor

@MattyMay MattyMay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a test to cover skipping subdirectories and fixed a typo. LGTM if my changes LGTY.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL .gitkeep

@awdeorio
Copy link
Contributor Author

awdeorio commented Nov 4, 2023

LGTM thanks for the addition

@awdeorio awdeorio merged commit a5ded12 into develop Nov 4, 2023
3 checks passed
@awdeorio awdeorio deleted the input-file branch November 4, 2023 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants