????JVM??????????
????JVM??????????????
????Java????????????????????????????????????????????????
?????????棺?????????
???????棺???????????洢??????????????
????????棺????????????ú??????
?????????????????????I/O????
???????????С???÷??????????Java??????????????????С??????JVM??????????????????????棬???????????JVM?????????
??????????С????????????????????????OOM????????
?????????????????????????????????????????????????????????????????????????????????JVM?????????????????????????????????
??????????CMS???????????Java???????滹????????????????ú????????????????????
????????????Java?????????????????????????????С????????Java??ó????????????????
?????????????????з?????Elasticsearch?У????????С??????128K?????256K?????Java7??Java6???????????? ??????????Java7????μ????????????????????????磬??????continuations??????????????????Continuations??????
????Э????????????green thread????????fiber????????? ????????????I/O????????????????????????????????????????д?????????????????????÷?????I/O??Elasticsearch???????????????Netty I/O????Guava??Elasticsearch?????????????????Java7?????????????????????????????
?????????????????С?????????????????????????????????????CPU???????????????JVM?汾????????????????????????????CPU????????????JVM???????С????????????????????г??????????????Sloaris Sparc 64λ??JVM Xss????512K??????и????????Sloaris X86?320K??Linux???256K??Windows 32λJava6???320K??Windows 64λ???1024K??
???????????
????????GB??????????????????????????????????????????G????????????????檔
????Java???????????????2006???Java6????????????????????????????????ж??????????????GC??????? stop - the - world??Ρ?CMS??????????????????????? ??????????GC?????????????????????????????????????????????Prateek Khanna ?? Aaron Morton??????CMS???????????????????????????
????????Stop-the-world???
???????????????Elasticsearch???????CMS????????????????????????????GC???????????????????????CMS???????????????????????????????Щ????????????????MB?????????飬?????????Щ??????????CMS????????????????????
????MB??????????????Lucene segment-based?????????????????????????????CMS???????????????Lucene?????ε??????????ò???index.merge.policy.segments_per_tier
??????????
??????????????????????????????Java JVM?????????????????????????????????????????????????????????????л?????????????????￡????????????????????涼???????????????????????????????????????????????????GC?????????????????????????????????????檔
??????????????????
??????Java JDK 7u4?????Garbage-First??G1?????????Java7????????????????????????????????????檔??????潵?????????????????????????????????????????????????????????????????в?????е????????????????????С??仯???ж????????????仯??
????G1?????????????????????????????????????????????￡????????е????
????1. ???????????????50%??Java??
????2. ??????????????promotion?????????仯
????3. ?????gc????compaction????????????0.5??1s??
?????????????G1????????????????????????????黹????????
????G1????????????????CPU??????????????????????????????????CPU????????????￡?CMS?????????
????????Elasticsearch?????G1??ζ????г?????stop-the-world??Σ????????????????????buffer memory????I/O????????????????????????????????С??????????????G1?????????CPU?????
??????????????
????????????????????????????????????Щ?????
????1. ????????????????????????????????????????
????2. ????κ????饗log everything???????????????????????????????????????
????3. ???????????????????????????????
????4. ?????????
????5. ????????????
????6. ????????ú????
????7. ???????????и??????????????????????ж?????????????
????Elasticsearch?????????????
????Elasticsearch?????GC??warns?????????????????
[2012-11-26 18:13:53??166][WARN ][monitor.jvm              ] [Ectokid] [gc][ParNew][1135087][11248] duration [2.6m]?? collections [1]/[2.7m]?? total [2.6m]/[6.8m]?? memory [2.4gb]->[2.3gb]/[3.8gb]?? all_pools {[Code Cache] [13.7mb]->[13.7mb]/[48mb]}{[Par Eden Space] [109.6mb]->[15.4mb]/[1gb]}{[Par Survivor Space] [136.5mb]->[0b]/[136.5mb]}{[CMS Old Gen] [2.1gb]->[2.3gb]/[2.6gb]}{[CMS Perm Gen] [35.1mb]->[34.9mb]/[82mb]}
????JvmMonitorService????????????÷????
Logfile Explanation
gc ?????е?gc
ParNew new parallel garbage collector
duration 2.6m gc????2.6????
collections [1]/[2.7m] ????????????????2.7????
memory [2.4gb]->[2.3gb]/[3.8gb] ???????? ?????2.4gb?? ??????2.3gb?? ????3.8gb???
Code Cache [13.7mb]->[13.7mb]/[48mb] code cache??????
Par Eden Space [109.6mb]->[15.4mb]/[1gb] Par Eden Space??????
Par Survivor Space [136.5mb]->[0b]/[136.5mb] Par Survivor Space??????
CMS Old Gen [2.1gb]->[2.3gb]/[2.6gb] CMS Old Gen??????
CMS Perm Gen [35.1mb]->[34.9mb]/[82mb] CMS Perm Gen??????
JvmMonitorSer
?????Щ????
????1. ?????Java 6u22????????汾????Elasticsearch??????淽???bug????Щ???????????bug?????????Elasticsearch?????????С?????OpenJDK 6?????????Sun/Oracle??汾????????????????bug??
????2. ????Java6?????Java7??Oracle????Java6???μ?2013??2????????????Elasticsearch???????????μ??????????????μ?????????????????????JVM?м?????????????????汾?????°汾????????????У??????????Java???л????????????
????3. ???????Java???л?????????????????Ρ?????sa????????????Java?汾??????Java???????????
????4. ??С????????Elasticsearch?????????п???????????????Elasticsearch?????????????????????????????????????????????3???????п????????
????5. ?????JVM???????????????????????????????????????????????????????????????????????????????????Elasticsearch?????????????С?????index.merge.policy.segments_per_tierparameter?????????ε?????
????6. ???????????????????????????????????????????
????7. ?????????????????????????????????С?????????
????8. ??????CMS??????????????????????????? -XX:CMSWaitDuration ??????
????9. ??????????6-8GB????????CMS???????????????????????????????stop-the-world??Σ????м?????????????CMSInitiatingOccupancyFraction????????????GC????????????С??????G1???????????
????10. ?????????????????????????????????г??????JVM?????java?????м???java -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version?????????