What's changing

Adding URL input support to allow users to fetch and clean content directly from websites. Changes include:

1. UI Changes (see app.py):

   • Added URL text input field
   • Added "Clean URL Content" button
   • Maintains visual hierarchy with file upload

2. Backend Implementation (see app.py):

   • Added URL content fetching using requests
   • Integrated with existing HTML cleaning pipeline
   • Added specific error handling for network and parsing issues

3. Testing (see test_url_input.py):

   • Added URL integration tests in test_url_input.py
   • Verified existing file upload tests in test_data_load_and_clean.py
   • Added error handling tests
   • Added content quality verification
   • Added size limit checks

4. Documentation Updates:

   • Updated step-by-step guide (see step-by-step-guide.md)
   • Updated README (see README.md)

5. Fixes and Improvements:

   • Restored file upload functionality that was accidentally removed
   • Fixed formatting issues in test files
   • Added proper error handling for URL processing
   • Improved state management with st.session_state['clean_text']
   • Fixed linting issues and code formatting

Closes #32

How to test it

1. Run the demo:

bash

python -m streamlit run demo/app.py

2. Test URL input:

   • Paste: https://blog.mozilla.ai/introducing-blueprints-customizable-ai-workflows-for-developers/
   • Click "Clean URL Content"
   • Verify raw and cleaned text display
   • Check cleaning quality (HTML removed, content preserved)

3. Test error handling:

   • Try invalid URL
   • Verify error message displays

Additional notes for reviewers

• Uses existing DATA_CLEANERS[".html"] for consistency with file upload
• Error handling matches Mozilla's error handling patterns
• Integration tests verify same cleaning quality as file upload
• Documentation updated to maintain consistency

I already...

• Tested in local environment and verified functionality
• Added integration tests in tests/integration/test_url_input.py
• Updated documentation in README and step-by-step guide
• Fixed linting issues and restored proper code formatting
• Verified both URL and file upload functionality work together
• Ran and passed all integration tests for both features