Getting Started

When using RxEnvironments, you only have to specify the dynamics of your environment. Let's create the Bayesian Thermostat environment in RxEnvironments. For this example, you need Distributions.jl installed in your environment as well.

Let's create the basics of our environment:

using RxEnvironments
using Distributions

# Empty agent, could contain states as well
struct ThermostatAgent end

mutable struct BayesianThermostat{T}
    temperature::T
    min_temp::T
    max_temp::T
end

# Helper functions
temperature(env::BayesianThermostat) = env.temperature
min_temp(env::BayesianThermostat) = env.min_temp
max_temp(env::BayesianThermostat) = env.max_temp
noise(env::BayesianThermostat) = Normal(0.0, 0.1)
set_temperature!(env::BayesianThermostat, temp::Real) = env.temperature = temp
function add_temperature!(env::BayesianThermostat, diff::Real)
    env.temperature += diff
    if temperature(env) < min_temp(env)
        set_temperature!(env, min_temp(env))
    elseif temperature(env) > max_temp(env)
        set_temperature!(env, max_temp(env))
    end
end

add_temperature! (generic function with 1 method)

By implementing RxEnvironments.receive!, RxEnvironments.what_to_send and RxEnvironments.update! for our environment, we can fully specify the behaviour of our environment, and RxEnvironments will take care of the rest. The RxEnvironments.receive! and RxEnvironments.what_to_send functions have a specific signature: RxEnvironments.receive!(receiver, emitter, action) takes as arguments the recipient of the action (in this example the environment), the emitter of the action (in this example the agent) and the action itself (in this example the change in temperature). The receive! function thus specifiec how an action from emitter to recipient affects the state of recipient. Always make sure to dispatch on the types of your environments, agents and actions, as RxEnvironments relies on Julia's multiple dispatch system to call the correct functions. Similarly for what_to_send, which takes the recipient and emitter (and potentially observation) as arguments, that computes the observation from emitter presented to recipient (when emitter has received observation). In our Bayesian Thermostat example, these functions look as follows:

# When the environment receives an action from the agent, we add the value of the action to the environment temperature.
RxEnvironments.receive!(recipient::BayesianThermostat, emitter::ThermostatAgent, action::Real) = add_temperature!(recipient, action)

# The environment sends a noisy observation of the temperature to the agent.
RxEnvironments.what_to_send(recipient::ThermostatAgent, emitter::BayesianThermostat) = temperature(emitter) + rand(noise(emitter))

# The environment cools down over time.
RxEnvironments.update!(env::BayesianThermostat, elapsed_time)= add_temperature!(env, -0.1 * elapsed_time)

Now we've fully specified our environment, and we can interact with it. In order to create the environment, we use the RxEnvironment struct, and we add an agent to this environment using add!:

environment = RxEnvironment(BayesianThermostat(0.0, -10.0, 10.0); emit_every_ms = 900)
agent = add!(environment, ThermostatAgent())

Continuous RxEntity{Main.ThermostatAgent}

Now we can have the agent conduct actions in our environment. Let's have the agent conduct some actions, and inspect the observations that are being returned by the environment:

# Subscribe a logger actor to the observations of the agent
RxEnvironments.subscribe_to_observations!(agent, RxEnvironments.logger())

# Conduct 10 actions:
for i in 1:10
    action = rand()
    RxEnvironments.send!(environment, agent, action)
    sleep(1)
end

[LogActor] Data: 0.7471371682870658
[LogActor] Data: 0.41841462813865554
[LogActor] Data: 1.0844422445079342
[LogActor] Data: 1.0476054151478535
[LogActor] Data: 1.1311701725470173
[LogActor] Data: 1.813328571784608
[LogActor] Data: 1.4416030836764542
[LogActor] Data: 2.0773447996949126
Error in Timer:
StackOverflowError()[LogActor] Data: 1.9790532679187103
[LogActor] Data: 2.558263490216912

Stacktrace:
 [1] error(s::StackOverflowError)
   @ Base ./error.jl:44
 [2] on_error!(actor::RxEnvironments.TimerActor{RxEnvironments.RxEntity{Main.__atexample__named__mountaincar.MountainCarEnvironment, RxEnvironments.ContinuousEntity, RxEnvironments.ActiveEntity, Any}}, err::StackOverflowError)
   @ RxEnvironments ~/work/RxEnvironments.jl/RxEnvironments.jl/src/timer.jl:111
 [3] error!(actor::RxEnvironments.TimerActor{RxEnvironments.RxEntity{Main.__atexample__named__mountaincar.MountainCarEnvironment, RxEnvironments.ContinuousEntity, RxEnvironments.ActiveEntity, Any}}, err::StackOverflowError)
   @ Rocket ~/.julia/packages/Rocket/LrFUI/src/actor.jl:223
 [4] on_error!(actor::Rocket.TimerActor{RxEnvironments.TimerActor{RxEnvironments.RxEntity{Main.__atexample__named__mountaincar.MountainCarEnvironment, RxEnvironments.ContinuousEntity, RxEnvironments.ActiveEntity, Any}}}, err::StackOverflowError)
   @ Rocket ~/.julia/packages/Rocket/LrFUI/src/observable/timer.jl:105
 [5] error!(actor::Rocket.TimerActor{RxEnvironments.TimerActor{RxEnvironments.RxEntity{Main.__atexample__named__mountaincar.MountainCarEnvironment, RxEnvironments.ContinuousEntity, RxEnvironments.ActiveEntity, Any}}}, err::StackOverflowError)
   @ Rocket ~/.julia/packages/Rocket/LrFUI/src/actor.jl:223
 [6] (::Rocket.var"#138#139"{Rocket.TimerActor{RxEnvironments.TimerActor{RxEnvironments.RxEntity{Main.__atexample__named__mountaincar.MountainCarEnvironment, RxEnvironments.ContinuousEntity, RxEnvironments.ActiveEntity, Any}}}})(timer::Timer)
   @ Rocket ~/.julia/packages/Rocket/LrFUI/src/observable/timer.jl:117
 [7] macro expansion
   @ ./asyncevent.jl:281 [inlined]
 [8] (::Base.var"#702#703"{Rocket.var"#138#139"{Rocket.TimerActor{RxEnvironments.TimerActor{RxEnvironments.RxEntity{Main.__atexample__named__mountaincar.MountainCarEnvironment, RxEnvironments.ContinuousEntity, RxEnvironments.ActiveEntity, Any}}}}, Timer})()
   @ Base ./task.jl:134[LogActor] Data: 2.4555913319785008
[LogActor] Data: 2.967617940976887
[LogActor] Data: 2.9246388605916054
[LogActor] Data: 3.0854381747698563
[LogActor] Data: 3.0480082091071274
[LogActor] Data: 4.0350884005225485
[LogActor] Data: 3.9836012624648025
[LogActor] Data: 4.987724232604934
[LogActor] Data: 4.721325554265061
[LogActor] Data: 4.873779782059621
[LogActor] Data: 4.737013867656735

Congratulations! You've now implemented a basic environment in RxEnvironments.