-
Notifications
You must be signed in to change notification settings - Fork 762
JVM Options
Miscellaneous notes on JVM options of interest to Heritrix operators. Eventually any wisdom here will find its way into the launch-script defaults; but new/untested/optional things may be discussed here.
Recommended:
-XX:+UseParallelOldGC
May be useful to speed full GC compactions on multi-core machines.See http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html#par_gc
-XX:+DoEscapeAnalysis
Available in 6u14, may improve performance; see http://java.sun.com/javase/6/webnotes/6u14.html
-XX:+UseCompressedOops
Available in 6u14, may improve memory use/GC if using 64-bit JVM on heaps <32GB in size; see http://java.sun.com/javase/6/webnotes/6u14.html
Maybe:
-XX:-UseGCOverheadLimit
Disables a OOME that is created not when memory is truly depleted, but when an excessive amount of time is spent in GC. We've seen this OOME at times when a requested memory-intensive report may have completed if not for this OOME (albeit in an extreme length of time).
Structured Guides:
User Guide
- Introduction
- New Features in 3.0 and 3.1
- Your First Crawl
- Checkpointing
- Main Console Page
- Profiles
- Heritrix Output
- Common Heritrix Use Cases
- Jobs
- Configuring Jobs and Profiles
- Processing Chains
- Credentials
- Creating Jobs and Profiles
- Outside the User Interface
- A Quick Guide to Creating a Profile
- Job Page
- Frontier
- Spring Framework
- Multiple Machine Crawling
- Heritrix3 on Mac OS X
- Heritrix3 on Windows
- Responsible Crawling
- Adding URIs mid-crawl
- Politeness parameters
- BeanShell Script For Downloading Video
- crawl manifest
- JVM Options
- Frontier queue budgets
- BeanShell User Notes
- Facebook and Twitter Scroll-down
- Deduping (Duplication Reduction)
- Force speculative embed URIs into single queue.
- Heritrix3 Useful Scripts
- How-To Feed URLs in bulk to a crawler
- MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
- WARC (Web ARChive)
- When taking a snapshot Heritrix renames crawl.log
- YouTube
- H3 Dev Notes for Crawl Operators
- Development Notes
- Spring Crawl Configuration
- Build Box
- Potential Cleanup-Refactorings
- Future Directions Brainstorming
- Documentation Wishlist
- Web Spam Detection for Heritrix
- Style Guide
- HOWTO Ship a Heritrix Release
- Heritrix in Eclipse