All states in the current belief are terminal

[yangyou95]

Hi!

Thanks for publishing the codes. I am currently running AdaOPS on various benchmarks.

However, on Bumper Roomba, when I run AdaOPS, sometimes this issue occurs and stops the planning programming:

All states in the current belief are terminal.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] AdaOPSTree(p::AdaOPSPlanner{Int64, RoombaAct, Bool, BumperPOMDP{DiscreteRoombaStateSpace, Vector{RoombaAct}, Int64}, 0, IndependentBounds{Float64, Float64}, POMDPTools.POMDPDistributions.Deterministic{Bool}, Random.MersenneTwister}, b0::ParticleFilters.ParticleCollection{Int64})
   @ AdaOPS ~/.julia/packages/AdaOPS/xXOEG/src/tree.jl:5
 [3] build_tree(p::AdaOPSPlanner{Int64, RoombaAct, Bool, BumperPOMDP{DiscreteRoombaStateSpace, Vector{RoombaAct}, Int64}, 0, IndependentBounds{Float64, Float64}, POMDPTools.POMDPDistributions.Deterministic{Bool}, Random.MersenneTwister}, b0::ParticleFilters.ParticleCollection{Int64})
   @ AdaOPS ~/.julia/packages/AdaOPS/xXOEG/src/planner.jl:2
 [4] action_info(p::AdaOPSPlanner{Int64, RoombaAct, Bool, BumperPOMDP{DiscreteRoombaStateSpace, Vector{RoombaAct}, Int64}, 0, IndependentBounds{Float64, Float64}, POMDPTools.POMDPDistributions.Deterministic{Bool}, Random.MersenneTwister}, b::ParticleFilters.ParticleCollection{Int64})
   @ AdaOPS ~/.julia/packages/AdaOPS/xXOEG/src/pomdps_glue.jl:7
 [5] iterate(it::POMDPTools.Simulators.POMDPSimIterator{(:s, :a, :o, :r), BumperPOMDP{DiscreteRoombaStateSpace, Vector{RoombaAct}, Int64}, AdaOPSPlanner{Int64, RoombaAct, Bool, BumperPOMDP{DiscreteRoombaStateSpace, Vector{RoombaAct}, Int64}, 0, IndependentBounds{Float64, Float64}, POMDPTools.POMDPDistributions.Deterministic{Bool}, Random.MersenneTwister}, ParticleFilters.BasicParticleFilter{BumperPOMDP{DiscreteRoombaStateSpace, Vector{RoombaAct}, Int64}, BumperPOMDP{DiscreteRoombaStateSpace, Vector{RoombaAct}, Int64}, ParticleFilters.LowVarianceResampler, Random.MersenneTwister, Vector{Int64}}, Random._GLOBAL_RNG, ParticleFilters.ParticleCollection{Int64}, Int64}, is::Tuple{Int64, Int64, ParticleFilters.ParticleCollection{Int64}})
   @ POMDPTools.Simulators ~/.julia/packages/POMDPTools/i3nmI/src/Simulators/stepthrough.jl:91
 [6] top-level scope
   @ ./In[18]:17
 [7] eval
   @ ./boot.jl:368 [inlined]
 [8] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
   @ Base ./loading.jl:1428

Basically, I run AdaOPS with following settings of Roomba:

# Init Bumper Roomba POMDP
using POMDPs, POMDPModels, POMDPSimulators, AdaOPS
using RoombaPOMDPs

# keep the same action set as in AdaOPS
max_speed = 5.0
speed_interval = 2.0
max_turn_rate = 1.0
turn_rate_interval = 1.0
action_space = vec([RoombaAct(v, om) for v in 1:speed_interval:max_speed, om in -max_turn_rate:turn_rate_interval:max_turn_rate])

# State Discretezation
num_x_pts = 25 # e.g. 41
num_y_pts = 16 # e.g. 26
num_th_pts = 10 # e.g. 20
sspace = DiscreteRoombaStateSpace(num_x_pts,num_y_pts,num_th_pts)
pomdp = RoombaPOMDP(sensor=Bumper(), mdp=RoombaMDP(config=3, aspace=action_space, v_max=max_speed,  sspace=sspace))

and I init AdaOPS as follows (BTW, I am not sure what values are used by AdaOPS on this domain for init L and U):

init_L = -20.0
init_U = 0.3
solver = AdaOPSSolver(bounds=IndependentBounds(init_L, init_U))
planner = solve(solver, pomdp) 
gamma = discount(pomdp)
sim_nb = 1000
max_depth = 100

avg_sum_r = 0.0
for i in 1:sim_nb
    temp_sum_r = 0.0
    step = 0
    for (s, a, o, r) in stepthrough(pomdp, planner, "s,a,o,r", max_steps=max_depth)
        temp_sum_r += (gamma^step)*r
        step += 1
    end
    avg_sum_r += temp_sum_r
end

I think the problem is caused by belief merging, as step goes on, it gradually falls to a wrong belief filled with terminal states.

I am not sure how to quickly fix this, if possible, could you let me know how can I configure AdaOPS on this domain so that this issue won't occur? For example, providing example codes for evaluation on Roomba will be really helpful.

Many thanks!

[yangyou95]

Or maybe just set a default_action in this case?

[Wu-Chenyang]

Hello,

I'm sorry for the inconvenience you've encountered while running AdaOPS on the Bumper Roomba benchmark. Based on your description, it appears that the issue you are experiencing is likely due to a failure in the particle filter's state tracking. This failure is causing all particles in the maintained belief to be terminal, while the real state is not.

In particle filter-based POMDP solvers, the accuracy of the belief state is crucial for the performance of the planning algorithm. If the particle filter does not maintain a diverse and representative set of particles, the belief may converge prematurely to a non-representative state (in this case, a terminal state). This problem can be exacerbated in environments with a lot of uncertainty or where the observation model does not provide enough information to correct inaccurate beliefs.

You might consider adjust the belief updater used in the simulation, increasing the number of particles and modifying the resampling strategy.

let l = 40, w = 25, t = 24, ζ=0.001, η=0.005, v_noise_coeff=0.02, om_noise_coeff=0.01, grid = StateGrid(range(-25, stop=15, length=l+1)[2:end-1],
                    range(-20, stop=5, length=w+1)[2:end-1],
                    range(0, stop=2*pi, length=t+1)[2:end-1])
    n_init = ceil(Int, KLDSampleSize(convert(Int, 11*l*w*t/20), ζ, η))
    global belief_updater = (m)->RoombaParticleFilter(m, n_init, v_noise_coeff, om_noise_coeff, KLDResampler(grid, n_init, ζ, η))
end

This is the belief updater I used in the BumperRoomba. You can give it a try. I hope it helps. Please let me know if you have any further questions or if there's anything else I can assist you with.