A helper class for benchmarking that controls running multiple queries. More...

#include <benchmarking.h>

Public Member Functions
PlanDataSetPtr	benchmark (std::size_t n_threads=1) const
	Run benchmarking on this experiment. Note that, for some planners, multiple threads cannot be used without polluting the dataset, due to reuse of underlying datastructures between queries, e.g., the robowflex_ompl planner. More...

Building Experiment
	Experiment (const std::string &name, const Profiler::Options &options, double allowed_time=60.0, std::size_t trials=100, bool timeout=false)
	Constructor. More...

void	addQuery (const std::string &planner_name, const SceneConstPtr &scene, const PlannerPtr &planner, const planning_interface::MotionPlanRequest &request)
	Add a query to the experiment for profiling. More...

void	addQuery (const std::string &planner_name, const SceneConstPtr &scene, const PlannerPtr &planner, const MotionRequestBuilder &request)
	Add a query to the experiment for profiling. More...

void	addQuery (const std::string &planner_name, const SceneConstPtr &scene, const PlannerPtr &planner, const MotionRequestBuilderPtr &request)
	Add a query to the experiment for profiling. More...

Configuration, Getters, Setters
Profiler::Options &	getOptions ()
	Get the options used by the underlying profiler. More...

Profiler &	getProfiler ()
	Get a reference to the profiler used by the experiment. Add callback functions to this profiler for custom metrics. More...

const Profiler &	getProfilerConst () const
	Get a reference to the profiler used by the experiment. More...

const std::vector< PlanningQuery > &	getQueries () const
	Get the queries added to this experiment. More...

void	enableMultipleRequests ()
	If called, will enable planners to use multiple threads. By default, planners are requested to only use one thread. More...

void	overridePlanningTime ()
	If called, instead of using global allotted planning time (provided in the constructor), the benchmark will use the requested planning time in the planning request of the query. More...

Private Attributes
const std::string	name_
	Name of this experiment. More...

double	allowed_time_
	Allotted time to use for each query. More...

std::size_t	trials_
	Number of trials to run each query for. More...

bool	timeout_

bool	enforce_single_thread_ {true}

bool	override_planning_time_ {true}

Profiler::Options	options_
	Options for profiler. More...

Profiler	profiler_
	Profiler to use for extracting data. More...

std::vector< PlanningQuery >	queries_
	Queries to test. More...

PreRunCallback	pre_callback_
	Pre-run callback. More...

PostRunCallback	post_callback_
	Post-run callback. More...

PostQueryCallback	complete_callback_
	Post-run callback with dataset. More...

Callback Functions
using	PreRunCallback = std::function< void(const PlanningQuery &query)>
	A callback function that is called before every query is profiled. More...

using	PostRunCallback = std::function< void(PlanData &result, const PlanningQuery &query)>
	A callback function that is called after every query is profiled. More...

using	PostQueryCallback = std::function< void(PlanDataSetPtr dataset, const PlanningQuery &query)>

void	setPreRunCallback (const PreRunCallback &callback)
	Set the pre-query callback function. More...

void	setPostRunCallback (const PostRunCallback &callback)
	Set the post-query callback function. More...

void	setPostQueryCallback (const PostQueryCallback &callback)
	Set the post-dataset callback function. More...

Detailed Description

A helper class for benchmarking that controls running multiple queries.

Definition at line 348 of file benchmarking.h.

Member Typedef Documentation

◆ PostQueryCallback

using robowflex::Experiment::PostQueryCallback = std::function<void(PlanDataSetPtr dataset, const PlanningQuery &query)>

A callback function that is called after a result is added to the dataset.

Parameters

[in,out]

Definition at line 457 of file benchmarking.h.

◆ PostRunCallback

using robowflex::Experiment::PostRunCallback = std::function<void(PlanData &result, const PlanningQuery &query)>

A callback function that is called after every query is profiled.

Parameters

[in,out]	result	The result computed by the profiler.
[in]	query	The query that was profiled.

Definition at line 451 of file benchmarking.h.

◆ PreRunCallback

using robowflex::Experiment::PreRunCallback = std::function<void(const PlanningQuery &query)>

A callback function that is called before every query is profiled.

Parameters

[in] query The query to be profiled.

Definition at line 445 of file benchmarking.h.

Constructor & Destructor Documentation

◆ Experiment()

Experiment::Experiment	(	const std::string &	name,
		const Profiler::Options &	options,
		double	allowed_time = `60.0`,
		std::size_t	trials = `100`,
		bool	timeout = `false`
	)

Constructor.

Parameters

[in]	name	Name of this experiment.
[in]	options	Options for the internal profiler.
[in]	allowed_time	Total planning time allowed for each query.
[in]	trials	Number of trials to run each query for.
[in]	timeout	If true, will re-run each query until the total time taken has exceeded the allotted time.

Experiment

Definition at line 308 of file benchmarking.cpp.

   : name_(name), allowed_time_(allowed_time), trials_(trials), timeout_(timeout), options_(options)
 {
 }

Member Function Documentation

◆ addQuery() [1/3]

void Experiment::addQuery	(	const std::string &	planner_name,
		const SceneConstPtr &	scene,
		const PlannerPtr &	planner,
		const MotionRequestBuilder &	request
	)

Add a query to the experiment for profiling.

Parameters

[in]	planner_name	Name to associate with this query. Does not need to be unique.
[in]	scene	Scene to use for query.
[in]	planner	Planner to use for query.
[in]	request	Request to use for query.

Definition at line 322 of file benchmarking.cpp.

 {
     addQuery(planner_name, scene, planner, request.getRequestConst());
 }

◆ addQuery() [2/3]

void Experiment::addQuery	(	const std::string &	planner_name,
		const SceneConstPtr &	scene,
		const PlannerPtr &	planner,
		const MotionRequestBuilderPtr &	request
	)

Add a query to the experiment for profiling.

Parameters

[in]	planner_name	Name to associate with this query. Does not need to be unique.
[in]	scene	Scene to use for query.
[in]	planner	Planner to use for query.
[in]	request	Request to use for query.

Definition at line 330 of file benchmarking.cpp.

 {
     addQuery(planner_name, scene, planner, *request);
 }

◆ addQuery() [3/3]

void Experiment::addQuery	(	const std::string &	planner_name,
		const SceneConstPtr &	scene,
		const PlannerPtr &	planner,
		const planning_interface::MotionPlanRequest &	request
	)

Add a query to the experiment for profiling.

Parameters

[in]	planner_name	Name to associate with this query. Does not need to be unique.
[in]	scene	Scene to use for query.
[in]	planner	Planner to use for query.
[in]	request	Request to use for query.

Definition at line 314 of file benchmarking.cpp.

 {
     queries_.emplace_back(planner_name, scene, planner, request);
 }

◆ benchmark()

PlanDataSetPtr Experiment::benchmark ( std::size_t n_threads = 1 ) const

Run benchmarking on this experiment. Note that, for some planners, multiple threads cannot be used without polluting the dataset, due to reuse of underlying datastructures between queries, e.g., the robowflex_ompl planner.

Parameters

[in] n_threads Number of threads to use for benchmarking.

Returns: The computed dataset.

Definition at line 383 of file benchmarking.cpp.

 {
     // Setup dataset to return
     auto dataset = std::make_shared<PlanDataSet>();
     dataset->name = name_;
     dataset->start = IO::getDate();
     dataset->allowed_time = allowed_time_;
     dataset->trials = trials_;
     dataset->enforced_single_thread = enforce_single_thread_;
     dataset->run_till_timeout = timeout_;
     dataset->threads = n_threads;
     dataset->queries = queries_;
  
     struct ThreadInfo
     {
         ThreadInfo() = default;
         ThreadInfo(const PlanningQuery *query, std::size_t trial, std::size_t index)
           : query(query), trial(trial), index(index)
         {
         }
  
         const PlanningQuery *query;
         std::size_t trial;
         std::size_t index;
     };
  
     std::queue<ThreadInfo> todo;
     std::mutex mutex;
  
     for (std::size_t i = 0; i < queries_.size(); ++i)
     {
         const auto &query = queries_[i];
  
         // Check if this name is unique, if so, add it to dataset list.
         const auto &it = std::find(dataset->query_names.begin(), dataset->query_names.end(), query.name);
         if (it == dataset->query_names.end())
             dataset->query_names.emplace_back(query.name);
  
         for (std::size_t j = 0; j < trials_; ++j)
             todo.emplace(&query, j, i);
     }
  
     std::vector<std::shared_ptr<std::thread>> threads;
     std::size_t completed_queries = 0;
     std::size_t total_queries = todo.size();
  
     for (std::size_t i = 0; i < n_threads; ++i)
         threads.emplace_back(std::make_shared<std::thread>([&]() {
             std::size_t id = IO::getThreadID();
             while (true)
             {
                 ThreadInfo info;
  
                 {
                     std::unique_lock<std::mutex> lock(mutex);
                     if (todo.empty())  // All done, exit.
                         return;
  
                     info = todo.front();
                     todo.pop();
                 }
  
                 RBX_INFO("[Thread %1%] Running Query %3% `%2%` Trial [%4%/%5%]",  //
                          id, info.query->name, info.index, info.trial + 1, trials_);
  
                 // If override, use global time. Else use query time.
                 double time_remaining =
                     (override_planning_time_) ? allowed_time_ : info.query->request.allowed_planning_time;
  
                 std::size_t timeout_trial = 0;
                 while (time_remaining > 0.)
                 {
                     planning_interface::MotionPlanRequest request = info.query->request;
                     request.allowed_planning_time = time_remaining;
  
                     if (enforce_single_thread_)
                         request.num_planning_attempts = 1;
  
                     // Call pre-run callbacks
                     info.query->planner->preRun(info.query->scene, request);
  
                     if (pre_callback_)
                         pre_callback_(*info.query);
  
                     // Profile query
                     auto data = std::make_shared<PlanData>();
                     profiler_.profilePlan(info.query->planner,  //
                                           info.query->scene,    //
                                           request,              //
                                           options_,             //
                                           *data);
  
                     // Add experiment specific metrics
                     data->metrics.emplace("query_trial", (int)info.trial);
                     data->metrics.emplace("query_index", (int)info.index);
                     data->metrics.emplace("query_timeout_trial", (int)timeout_trial);
                     data->metrics.emplace("query_start_time", IO::getSeconds(dataset->start, data->start));
                     data->metrics.emplace("query_finish_time", IO::getSeconds(dataset->start, data->finish));
  
                     data->query.name = log::format("%1%:%2%:%3%", info.query->name, info.trial, info.index);
  
                     if (timeout_)
                         data->query.name = data->query.name + log::format(":%4%", timeout_trial);
  
                     if (post_callback_)
                         post_callback_(*data, *info.query);
  
                     {
                         std::unique_lock<std::mutex> lock(mutex);
  
                         dataset->addDataPoint(info.query->name, data);
                         completed_queries++;
  
                         if (complete_callback_)
                             complete_callback_(dataset, *info.query);
                     }
  
                     if (timeout_)
                     {
                         time_remaining -= data->time;
                         RBX_INFO(                                                                           //
                             "[Thread %1%] Running Query %3% `%2%` till timeout, %4% seconds remaining...",  //
                             id, info.query->name, info.index, time_remaining);
                         timeout_trial++;
                     }
                     else
                         time_remaining = 0;
                 }
  
                 RBX_INFO("[Thread %1%] Completed Query %3% `%2%` Trial [%4%/%5%] Total: [%6%/%7%]",  //
                          id, info.query->name, info.index,                                           //
                          info.trial + 1, trials_,                                                    //
                          completed_queries, total_queries);
             }
         }));
  
     for (const auto &thread : threads)
         thread->join();
  
     dataset->finish = IO::getDate();
     dataset->time = IO::getSeconds(dataset->start, dataset->finish);
  
     return dataset;
 }

◆ enableMultipleRequests()

void Experiment::enableMultipleRequests ( )

If called, will enable planners to use multiple threads. By default, planners are requested to only use one thread.

Definition at line 358 of file benchmarking.cpp.

 {
     enforce_single_thread_ = false;
 }

◆ getOptions()

Profiler::Options & Experiment::getOptions ( )

Get the options used by the underlying profiler.

Returns: A reference to the options used.

Definition at line 338 of file benchmarking.cpp.

 {
     return options_;
 }

◆ getProfiler()

Profiler & Experiment::getProfiler ( )

Get a reference to the profiler used by the experiment. Add callback functions to this profiler for custom metrics.

Returns: The profiler.

Definition at line 343 of file benchmarking.cpp.

 {
     return profiler_;
 }

◆ getProfilerConst()

const Profiler & Experiment::getProfilerConst ( ) const

Get a reference to the profiler used by the experiment.

Returns: The profiler.

Definition at line 348 of file benchmarking.cpp.

 {
     return profiler_;
 }

◆ getQueries()

const std::vector< PlanningQuery > & Experiment::getQueries ( ) const

Get the queries added to this experiment.

Returns: The queries added to the experiment.

Definition at line 353 of file benchmarking.cpp.

 {
     return queries_;
 }

◆ overridePlanningTime()

void Experiment::overridePlanningTime ( )

If called, instead of using global allotted planning time (provided in the constructor), the benchmark will use the requested planning time in the planning request of the query.

Definition at line 363 of file benchmarking.cpp.

 {
     override_planning_time_ = false;
 }

◆ setPostQueryCallback()

void Experiment::setPostQueryCallback ( const PostQueryCallback & callback )

Set the post-dataset callback function.

Parameters

[in] callback Callback to use.

Definition at line 378 of file benchmarking.cpp.

 {
     complete_callback_ = callback;
 }

◆ setPostRunCallback()

void Experiment::setPostRunCallback ( const PostRunCallback & callback )

Set the post-query callback function.

Parameters

[in] callback Callback to use.

Definition at line 373 of file benchmarking.cpp.

 {
     post_callback_ = callback;
 }

◆ setPreRunCallback()

void Experiment::setPreRunCallback ( const PreRunCallback & callback )

Set the pre-query callback function.

Parameters

[in] callback Callback to use.

Definition at line 368 of file benchmarking.cpp.

 {
     pre_callback_ = callback;
 }

Member Data Documentation

◆ allowed_time_

double robowflex::Experiment::allowed_time_

private

Allotted time to use for each query.

Definition at line 486 of file benchmarking.h.

◆ complete_callback_

PostQueryCallback robowflex::Experiment::complete_callback_

private

Post-run callback with dataset.

Definition at line 501 of file benchmarking.h.

◆ enforce_single_thread_

bool robowflex::Experiment::enforce_single_thread_ {true}

private

If true, will request each planner to only use a single thread.

Definition at line 490 of file benchmarking.h.

◆ name_

const std::string robowflex::Experiment::name_

private

Name of this experiment.

Definition at line 485 of file benchmarking.h.

◆ options_

Profiler::Options robowflex::Experiment::options_

private

Options for profiler.

Definition at line 495 of file benchmarking.h.

◆ override_planning_time_

bool robowflex::Experiment::override_planning_time_ {true}

private

If true, will override request planning time with global allowed time.

Definition at line 492 of file benchmarking.h.

◆ post_callback_

PostRunCallback robowflex::Experiment::post_callback_

private

Post-run callback.

Definition at line 500 of file benchmarking.h.

◆ pre_callback_

PreRunCallback robowflex::Experiment::pre_callback_

private

Pre-run callback.

Definition at line 499 of file benchmarking.h.

◆ profiler_

Profiler robowflex::Experiment::profiler_

private

Profiler to use for extracting data.

Definition at line 496 of file benchmarking.h.

◆ queries_

std::vector<PlanningQuery> robowflex::Experiment::queries_

private

Queries to test.

Definition at line 497 of file benchmarking.h.

◆ timeout_

bool robowflex::Experiment::timeout_

private

If true, will re-run planners on queries until total time taken has exceeded the allotted time.

Definition at line 488 of file benchmarking.h.

◆ trials_

std::size_t robowflex::Experiment::trials_

private

Number of trials to run each query for.

Definition at line 487 of file benchmarking.h.

The documentation for this class was generated from the following files:

Public Member Functions

Private Attributes

Callback Functions

Detailed Description

Member Typedef Documentation

◆ PostQueryCallback

◆ PostRunCallback

◆ PreRunCallback

Constructor & Destructor Documentation

◆ Experiment()

Member Function Documentation

◆ addQuery() [1/3]

◆ addQuery() [2/3]

◆ addQuery() [3/3]

◆ benchmark()

◆ enableMultipleRequests()

◆ getOptions()

◆ getProfiler()

◆ getProfilerConst()

◆ getQueries()

◆ overridePlanningTime()

◆ setPostQueryCallback()

◆ setPostRunCallback()

◆ setPreRunCallback()

Member Data Documentation

◆ allowed_time_

◆ complete_callback_

◆ enforce_single_thread_

◆ name_

◆ options_

◆ override_planning_time_

◆ post_callback_

◆ pre_callback_

◆ profiler_

◆ queries_

◆ timeout_

◆ trials_