Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

metrics collection gets panic #425

Open
vijaykanthm28 opened this issue Mar 24, 2017 · 4 comments
Open

metrics collection gets panic #425

vijaykanthm28 opened this issue Mar 24, 2017 · 4 comments
Assignees
@suganyakaliyamoorthy

Comments

@vijaykanthm28
Copy link
Member

Mar 24 09:04:03 vds0 virtengine[9620]: fatal error: concurrent map iteration and map write
Mar 24 09:04:03 vds0 virtengine[9620]: goroutine 6903 [running]:
Mar 24 09:04:03 vds0 virtengine[9620]: runtime.throw(0xd9a07c, 0x26)
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/golang/go/src/runtime/panic.go:596 +0x95 fp=0xc420edfbe0 sp=0xc420edfbc0
Mar 24 09:04:03 vds0 virtengine[9620]: runtime.mapiternext(0xc420edfd88)
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/golang/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc420edfc90 sp=0xc420edfbe0
Mar 24 09:04:03 vds0 virtengine[9620]: github.com/megamsys/vertice/metrix.eventSkews(0xc420698a00, 0x12, 0xc420c62000, 0xc4200a0448, 0xc4200a03e8)
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/.go/src/github.com/megamsys/vertice/metrix/scylla_output.go:67 +0x35e fp=0xc420edfdf8 sp=0xc420edfc90
Mar 24 09:04:03 vds0 virtengine[9620]: github.com/megamsys/vertice/metrix.(*OpenNebula).DeductBill(0xc420556050, 0xc420cba000, 0xc4208ba900, 0xc000)
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/.go/src/github.com/megamsys/vertice/metrix/one.go:56 +0x3fc fp=0xc420edfec8 sp=0xc420edfdf8
Mar 24 09:04:03 vds0 virtengine[9620]: github.com/megamsys/vertice/metrix.(*OpenNebula).Collect(0xc420556050, 0xc420cba000, 0x3, 0xc420aa9560)
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/.go/src/github.com/megamsys/vertice/metrix/one.go:74 +0xcb fp=0xc420edff10 sp=0xc420edfec8
Mar 24 09:04:03 vds0 virtengine[9620]: github.com/megamsys/vertice/metrix.(*MetricHandler).Collect(0x13be0e0, 0x13709a0, 0xc420556050, 0x0, 0x0, 0x0, 0xc420aa94a0, 0x8b2482)
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/.go/src/github.com/megamsys/vertice/metrix/metric_handler.go:13 +0x97 fp=0xc420edff58 sp=0xc420edff10
Mar 24 09:04:03 vds0 virtengine[9620]: github.com/megamsys/vertice/subd/metricsd.(*Handler).processCollector(0xc42013e1b8, 0x13be0e0, 0xc420222000, 0x13709a0, 0xc420556050, 0xc42086f1d8, 0xc3e320)
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/.go/src/github.com/megamsys/vertice/subd/metricsd/handler.go:16 +0x43 fp=0xc420edffa8 sp=0xc420edff58
Mar 24 09:04:03 vds0 virtengine[9620]: runtime.goexit()
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/golang/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420edffb0 sp=0xc420edffa8
Mar 24 09:04:03 vds0 virtengine[9620]: created by github.com/megamsys/vertice/subd/metricsd.(*Service).onedCollectors
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/.go/src/github.com/megamsys/vertice/subd/metricsd/service.go:128 +0x491
Mar 24 09:04:03 vds0 virtengine[9620]: goroutine 1 [chan receive, 47 minutes]:
Mar 24 09:04:03 vds0 virtengine[9620]: github.com/megamsys/vertice/cmd/vertice/run.(*Start).Run(0xc4201b8e40, 0xc420143540, 0x0, 0x0)
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/.go/src/github.com/megamsys/vertice/cmd/vertice/run/command.go:85 +0x5e3
Mar 24 09:04:03 vds0 virtengine[9620]: github.com/megamsys/libgo/cmd.(*Manager).Run(0xc4201ac160, 0xc420010130, 0x2, 0x2)
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/.go/src/github.com/megamsys/libgo/cmd/cmd.go:183 +0x15b7
Mar 24 09:04:03 vds0 virtengine[9620]: main.main()
Mar 24 09:04:03 vds0 virtengine[9620]: /home/jonathan/code/detio/gitpackager/vertice/build/xenial/virtengine/src/github.com/megamsys/virtengine/cmd/vertice/main.go:52 +0xb8
@rajesh-rajagopal
Copy link
Member

CUST : Vertice automatically stopped.
Panic error in Raid server

time="2017-04-16T20:17:50Z" level=debug msg="Request [POST] ==> http://167.114.244.54:9000/v2/balances/bill"
Apr 16 20:17:50 mirror virtengine[19647]: time="2017-04-16T20:17:50Z" level=debug msg="[Body] ({"id":"BAL7247771431765712102","account_id":"[email protected]","credit":"-75.775672","json_claz":"Megam::Balances","created_at":"2017-04-16T20:17:50.442328048Z","updated_at":"2017-04-16T20:17:50.442328235Z"})"Apr 16 20:17:50 mirror virtengine[19647]: fatal error: concurrent map read and map write
Apr 16 20:17:50 mirror virtengine[19647]: goroutine 119481 [running]:
Apr 16 20:17:50 mirror virtengine[19647]: runtime.throw(0xd960d8, 0x21)
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/software/go/src/runtime/panic.go:596 +0x95 fp=0xc425121d98 sp=0xc425121d78
Apr 16 20:17:50 mirror virtengine[19647]: runtime.mapaccess1_faststr(0xc3f040, 0xc420d40030, 0xd7b639, 0x7, 0x8)
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/software/go/src/runtime/hashmap_fast.go:217 +0x4cf fp=0xc425121df8 sp=0xc425121d
Apr 16 20:17:50 mirror virtengine[19647]: github.com/megamsys/vertice/metrix.(*OpenNebula).DeductBill(0xc42022e320, 0xc422a5a000, 0xc42230ef40
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/code/megam/workspace/go/src/github.com/megamsys/vertice/metrix/one.go:41 +0x12c
Apr 16 20:17:50 mirror virtengine[19647]: github.com/megamsys/vertice/metrix.(*OpenNebula).Collect(0xc42022e320, 0xc422a5a000, 0x3, 0x20002)
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/code/megam/workspace/go/src/github.com/megamsys/vertice/metrix/one.go:74 +0xcb f
Apr 16 20:17:50 mirror virtengine[19647]: github.com/megamsys/vertice/metrix.(*MetricHandler).Collect(0x13bf0e0, 0x13719a0, 0xc42022e320, 0x0,
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/code/megam/workspace/go/src/github.com/megamsys/vertice/metrix/metric_handler.go
Apr 16 20:17:50 mirror virtengine[19647]: github.com/megamsys/vertice/subd/metricsd.(*Handler).processCollector(0xc420166138, 0x13bf0e0, 0xc42
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/code/megam/workspace/go/src/github.com/megamsys/vertice/subd/metricsd/handler.go
Apr 16 20:17:50 mirror virtengine[19647]: runtime.goexit()
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/software/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc425121fb0 sp=0xc425121fa8
Apr 16 20:17:50 mirror virtengine[19647]: created by github.com/megamsys/vertice/subd/metricsd.(*Service).onedCollectors
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/code/megam/workspace/go/src/github.com/megamsys/vertice/subd/metricsd/service.go
Apr 16 20:17:50 mirror virtengine[19647]: goroutine 1 [chan receive, 4940 minutes]:
Apr 16 20:17:50 mirror virtengine[19647]: github.com/megamsys/vertice/cmd/vertice/run.(*Start).Run(0xc42016a9a0, 0xc4201765a0, 0x0, 0x0)
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/code/megam/workspace/go/src/github.com/megamsys/vertice/cmd/vertice/run/command.
Apr 16 20:17:50 mirror virtengine[19647]: github.com/megamsys/libgo/cmd.(*Manager).Run(0xc4201aa000, 0xc42006c130, 0x2, 0x2)
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/code/megam/workspace/go/src/github.com/megamsys/libgo/cmd/cmd.go:183 +0x15b7
Apr 16 20:17:50 mirror virtengine[19647]: main.main()
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/code/megam/workspace/go/src/github.com/megamsys/vertice/cmd/vertice/main.go:52 +
Apr 16 20:17:50 mirror virtengine[19647]: goroutine 17 [syscall, 4940 minutes, locked to thread]:
Apr 16 20:17:50 mirror virtengine[19647]: runtime.goexit()
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/software/go/src/runtime/asm_amd64.s:2197 +0x1
Apr 16 20:17:50 mirror virtengine[19647]: goroutine 19 [syscall, 4940 minutes]:
Apr 16 20:17:50 mirror virtengine[19647]: os/signal.signal_recv(0x0)
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/software/go/src/runtime/sigqueue.go:116 +0x104
Apr 16 20:17:50 mirror virtengine[19647]: os/signal.loop()
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/software/go/src/os/signal/signal_unix.go:22 +0x22
Apr 16 20:17:50 mirror virtengine[19647]: created by os/signal.init.1
Apr 16 20:17:50 mirror virtengine[19647]: /home/vijay/software/go/src/os/signal/signal_unix.go:28 +0x41

@indykish
Copy link
Member

@suganyakaliyamoorthy Good luck, lets see some pull req on this issue.

@rajesh-rajagopal
Copy link
Member

Vetice stopped in raid1 server.

Apr 17 22:33:33 mirror virtengine[21599]: time="2017-04-17T22:33:33Z" level=debug msg=" > [one-go] ( ´ ▽ ) SUCCESS" Apr 17 22:33:33 mirror virtengine[21599]: time="2017-04-17T22:33:33Z" level=debug msg="showback (1492467813, 1492468413) OK" Apr 17 22:33:33 mirror virtengine[21599]: time="2017-04-17T22:33:33Z" level=debug msg=" > [one-go] ( ´ ▽ ) SUCCESS"
Apr 17 22:33:33 mirror virtengine[21599]: time="2017-04-17T22:33:33Z" level=debug msg="showback (1492467813, 1492468413) OK"
Apr 17 22:33:33 mirror virtengine[21599]: fatal error: concurrent map writes
Apr 17 22:33:33 mirror virtengine[21599]: goroutine 17778 [running]:
Apr 17 22:33:33 mirror virtengine[21599]: runtime.throw(0xd8eba1, 0x15)
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/golang/go/src/runtime/panic.go:596 +0x95 fp=0xc420ec9d58 sp=0xc420ec9d38
Apr 17 22:33:33 mirror virtengine[21599]: runtime.mapassign(0xc428a0, 0xc420906000, 0xc420ec9ea8, 0x2)
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/golang/go/src/runtime/hashmap.go:499 +0x667 fp=0xc420ec9df8 sp=0xc420ec9d58
Apr 17 22:33:33 mirror virtengine[21599]: github.com/megamsys/vertice/metrix.(*OpenNebula).DeductBill(0xc4201ce050, 0xc420907530, 0xc420506f80
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/.go/src/github.com/megamsys/vertice/metrix/one.go:36 +0xb9 fp=0xc420ec9ec8 sp
Apr 17 22:33:33 mirror virtengine[21599]: github.com/megamsys/vertice/metrix.(*OpenNebula).Collect(0xc4201ce050, 0xc420907530, 0x3, 0x6efa7a)
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/.go/src/github.com/megamsys/vertice/metrix/one.go:74 +0xcb fp=0xc420ec9f10 sp
Apr 17 22:33:33 mirror virtengine[21599]: github.com/megamsys/vertice/metrix.(*MetricHandler).Collect(0x13c5140, 0x1377a00, 0xc4201ce050, 0x6f
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/.go/src/github.com/megamsys/vertice/metrix/metric_handler.go:13 +0x97 fp=0xc4
Apr 17 22:33:33 mirror virtengine[21599]: github.com/megamsys/vertice/subd/metricsd.(*Handler).processCollector(0xc42000e148, 0x13c5140, 0xc42
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/.go/src/github.com/megamsys/vertice/subd/metricsd/handler.go:16 +0x43 fp=0xc4
Apr 17 22:33:33 mirror virtengine[21599]: runtime.goexit()
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/golang/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420ec9fb0 sp=0xc420ec9fa8
Apr 17 22:33:33 mirror virtengine[21599]: created by github.com/megamsys/vertice/subd/metricsd.(*Service).onedCollectors
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/.go/src/github.com/megamsys/vertice/subd/metricsd/service.go:128 +0x491
Apr 17 22:33:33 mirror virtengine[21599]: goroutine 1 [chan receive, 700 minutes]:
Apr 17 22:33:33 mirror virtengine[21599]: github.com/megamsys/vertice/cmd/vertice/run.(*Start).Run(0xc42000ca40, 0xc420014690, 0x0, 0x0)
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/.go/src/github.com/megamsys/vertice/cmd/vertice/run/command.go:85 +0x5e3
Apr 17 22:33:33 mirror virtengine[21599]: github.com/megamsys/libgo/cmd.(*Manager).Run(0xc4201a6000, 0xc42006e130, 0x2, 0x2)
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/.go/src/github.com/megamsys/libgo/cmd/cmd.go:183 +0x15b7
Apr 17 22:33:33 mirror virtengine[21599]: main.main()
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/code/detio/gitpackager/vertice/build/xenial/virtengine/src/github.com/megamsy
Apr 17 22:33:33 mirror virtengine[21599]: goroutine 17 [syscall, 700 minutes, locked to thread]:
Apr 17 22:33:33 mirror virtengine[21599]: runtime.goexit()
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/golang/go/src/runtime/asm_amd64.s:2197 +0x1
Apr 17 22:33:33 mirror virtengine[21599]: goroutine 20 [syscall, 700 minutes]:
Apr 17 22:33:33 mirror virtengine[21599]: os/signal.signal_recv(0x0)
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/golang/go/src/runtime/sigqueue.go:116 +0x104
Apr 17 22:33:33 mirror virtengine[21599]: os/signal.loop()
Apr 17 22:33:33 mirror virtengine[21599]: /home/jonathan/golang/go/src/os/signal/signal_unix.go:22 +0x22
Apr 17 22:33:33 mirror virtengine[21599]: created by os/signal.init.1

@indykish
Copy link
Member

Is this sev 1 issue ?

@indykish indykish reopened this Apr 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@suganyakaliyamoorthy suganyakaliyamoorthy

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@indykish @vijaykanthm28 @rajesh-rajagopal @suganyakaliyamoorthy