Intel(R) Threading Building Blocks Doxygen Documentation  version 4.2.3
tbb::internal::generic_scheduler Class Referenceabstract

Work stealing task scheduler. More...

#include <scheduler.h>

Inheritance diagram for tbb::internal::generic_scheduler:
Collaboration diagram for tbb::internal::generic_scheduler:

Public Member Functions

bool is_task_pool_published () const
 
bool is_local_task_pool_quiescent () const
 
bool is_quiescent_local_task_pool_empty () const
 
bool is_quiescent_local_task_pool_reset () const
 
void attach_mailbox (affinity_id id)
 
void init_stack_info ()
 Sets up the data necessary for the stealing limiting heuristics. More...
 
bool can_steal ()
 Returns true if stealing is allowed. More...
 
void publish_task_pool ()
 Used by workers to enter the task pool. More...
 
void leave_task_pool ()
 Leave the task pool. More...
 
void reset_task_pool_and_leave ()
 Resets head and tail indices to 0, and leaves task pool. More...
 
task ** lock_task_pool (arena_slot *victim_arena_slot) const
 Locks victim's task pool, and returns pointer to it. The pointer can be NULL. More...
 
void unlock_task_pool (arena_slot *victim_arena_slot, task **victim_task_pool) const
 Unlocks victim's task pool. More...
 
void acquire_task_pool () const
 Locks the local task pool. More...
 
void release_task_pool () const
 Unlocks the local task pool. More...
 
taskprepare_for_spawning (task *t)
 Checks if t is affinitized to another thread, and if so, bundles it as proxy. More...
 
void commit_spawned_tasks (size_t new_tail)
 Makes newly spawned tasks visible to thieves. More...
 
void commit_relocated_tasks (size_t new_tail)
 Makes relocated tasks visible to thieves and releases the local task pool. More...
 
taskget_task (__TBB_ISOLATION_EXPR(isolation_tag isolation))
 Get a task from the local pool. More...
 
taskget_task (size_t T)
 Get a task from the local pool at specified location T. More...
 
taskget_mailbox_task (__TBB_ISOLATION_EXPR(isolation_tag isolation))
 Attempt to get a task from the mailbox. More...
 
tasksteal_task (__TBB_ISOLATION_EXPR(isolation_tag isolation))
 Attempts to steal a task from a randomly chosen thread/scheduler. More...
 
tasksteal_task_from (__TBB_ISOLATION_ARG(arena_slot &victim_arena_slot, isolation_tag isolation))
 Steal task from another scheduler's ready pool. More...
 
size_t prepare_task_pool (size_t n)
 Makes sure that the task pool can accommodate at least n more elements. More...
 
bool cleanup_master (bool blocking_terminate)
 Perform necessary cleanup when a master thread stops using TBB. More...
 
void assert_task_pool_valid () const
 
void attach_arena (arena *, size_t index, bool is_master)
 
void nested_arena_entry (arena *, size_t)
 
void nested_arena_exit ()
 
void wait_until_empty ()
 
void spawn (task &first, task *&next) __TBB_override
 For internal use only. More...
 
void spawn_root_and_wait (task &first, task *&next) __TBB_override
 For internal use only. More...
 
void enqueue (task &, void *reserved) __TBB_override
 For internal use only. More...
 
void local_spawn (task *first, task *&next)
 
void local_spawn_root_and_wait (task *first, task *&next)
 
virtual void local_wait_for_all (task &parent, task *child)=0
 
void free_scheduler ()
 Destroy and deallocate this scheduler object. More...
 
taskallocate_task (size_t number_of_bytes, __TBB_CONTEXT_ARG(task *parent, task_group_context *context))
 Allocate task object, either from the heap or a free list. More...
 
template<free_task_hint h>
void free_task (task &t)
 Put task on free list. More...
 
void deallocate_task (task &t)
 Return task object to the memory allocator. More...
 
bool is_worker () const
 True if running on a worker thread, false otherwise. More...
 
bool outermost_level () const
 True if the scheduler is on the outermost dispatch level. More...
 
bool master_outermost_level () const
 True if the scheduler is on the outermost dispatch level in a master thread. More...
 
bool worker_outermost_level () const
 True if the scheduler is on the outermost dispatch level in a worker thread. More...
 
unsigned max_threads_in_arena ()
 Returns the concurrency limit of the current arena. More...
 
virtual taskreceive_or_steal_task (__TBB_ISOLATION_ARG(__TBB_atomic reference_count &completion_ref_count, isolation_tag isolation))=0
 Try getting a task from other threads (via mailbox, stealing, FIFO queue, orphans adoption). More...
 
void free_nonlocal_small_task (task &t)
 Free a small task t that that was allocated by a different scheduler. More...
 
- Public Member Functions inherited from tbb::internal::scheduler
virtual void wait_for_all (task &parent, task *child)=0
 For internal use only. More...
 
virtual ~scheduler ()=0
 Pure virtual destructor;. More...
 

Static Public Member Functions

static bool is_version_3_task (task &t)
 
static bool is_proxy (const task &t)
 True if t is a task_proxy. More...
 
static generic_schedulercreate_master (arena *a)
 Initialize a scheduler for a master thread. More...
 
static generic_schedulercreate_worker (market &m, size_t index)
 Initialize a scheduler for a worker thread. More...
 
static void cleanup_worker (void *arg, bool worker)
 Perform necessary cleanup when a worker thread finishes. More...
 
static taskplugged_return_list ()
 Special value used to mark my_return_list as not taking any more entries. More...
 

Public Attributes

uintptr_t my_stealing_threshold
 Position in the call stack specifying its maximal filling when stealing is still allowed. More...
 
marketmy_market
 The market I am in. More...
 
FastRandom my_random
 Random number generator used for picking a random victim from which to steal. More...
 
taskmy_free_list
 Free list of small tasks that can be reused. More...
 
taskmy_dummy_task
 Fake root task created by slave threads. More...
 
long my_ref_count
 Reference count for scheduler. More...
 
bool my_auto_initialized
 True if *this was created by automatic TBB initialization. More...
 
__TBB_atomic intptr_t my_small_task_count
 Number of small tasks that have been allocated by this scheduler. More...
 
taskmy_return_list
 List of small tasks that have been returned to this scheduler by other schedulers. More...
 
- Public Attributes inherited from tbb::internal::intrusive_list_node
intrusive_list_nodemy_prev_node
 
intrusive_list_nodemy_next_node
 
- Public Attributes inherited from tbb::internal::scheduler_state
size_t my_arena_index
 Index of the arena slot the scheduler occupies now, or occupied last time. More...
 
arena_slotmy_arena_slot
 Pointer to the slot in the arena we own at the moment. More...
 
arenamy_arena
 The arena that I own (if master) or am servicing at the moment (if worker) More...
 
taskmy_innermost_running_task
 Innermost task whose task::execute() is running. A dummy task on the outermost level. More...
 
mail_inbox my_inbox
 
affinity_id my_affinity_id
 The mailbox id assigned to this scheduler. More...
 
scheduler_properties my_properties
 

Static Public Attributes

static const size_t quick_task_size = 256-task_prefix_reservation_size
 If sizeof(task) is <=quick_task_size, it is handled on a free list instead of malloc'd. More...
 
static const size_t null_arena_index = ~size_t(0)
 
static const size_t min_task_pool_size = 64
 

Protected Member Functions

 generic_scheduler (market &)
 

Friends

template<typename SchedulerTraits >
class custom_scheduler
 

Detailed Description

Work stealing task scheduler.

None of the fields here are ever read or written by threads other than the thread that creates the instance.

Class generic_scheduler is an abstract base class that contains most of the scheduler, except for tweaks specific to processors and tools (e.g. VTune). The derived template class custom_scheduler<SchedulerTraits> fills in the tweaks.

Definition at line 124 of file scheduler.h.

Constructor & Destructor Documentation

◆ generic_scheduler()

tbb::internal::generic_scheduler::generic_scheduler ( market m)
protected

Definition at line 88 of file scheduler.cpp.

89  : my_market(&m)
90  , my_random(this)
91  , my_ref_count(1)
92  , my_small_task_count(1) // Extra 1 is a guard reference
93 #if __TBB_SURVIVE_THREAD_SWITCH && TBB_USE_ASSERT
94  , my_cilk_state(cs_none)
95 #endif /* __TBB_SURVIVE_THREAD_SWITCH && TBB_USE_ASSERT */
96 {
97  __TBB_ASSERT( !my_arena_index, "constructor expects the memory being zero-initialized" );
98  __TBB_ASSERT( governor::is_set(NULL), "scheduler is already initialized for this thread" );
99 
100  my_innermost_running_task = my_dummy_task = &allocate_task( sizeof(task), __TBB_CONTEXT_ARG(NULL, &the_dummy_context) );
101 #if __TBB_PREVIEW_CRITICAL_TASKS
102  my_properties.has_taken_critical_task = false;
103 #endif
104  my_properties.outermost = true;
105 #if __TBB_TASK_PRIORITY
106  my_ref_top_priority = &m.my_global_top_priority;
107  my_ref_reload_epoch = &m.my_global_reload_epoch;
108 #endif /* __TBB_TASK_PRIORITY */
109 #if __TBB_TASK_GROUP_CONTEXT
110  // Sync up the local cancellation state with the global one. No need for fence here.
111  my_context_state_propagation_epoch = the_context_state_propagation_epoch;
112  my_context_list_head.my_prev = &my_context_list_head;
113  my_context_list_head.my_next = &my_context_list_head;
114  ITT_SYNC_CREATE(&my_context_list_mutex, SyncType_Scheduler, SyncObj_ContextsList);
115 #endif /* __TBB_TASK_GROUP_CONTEXT */
116  ITT_SYNC_CREATE(&my_dummy_task->prefix().ref_count, SyncType_Scheduler, SyncObj_WorkerLifeCycleMgmt);
117  ITT_SYNC_CREATE(&my_return_list, SyncType_Scheduler, SyncObj_TaskReturnList);
118 }
#define __TBB_CONTEXT_ARG(arg1, context)
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
#define ITT_SYNC_CREATE(obj, type, name)
Definition: itt_notify.h:123
FastRandom my_random
Random number generator used for picking a random victim from which to steal.
Definition: scheduler.h:162
task * my_innermost_running_task
Innermost task whose task::execute() is running. A dummy task on the outermost level.
Definition: scheduler.h:81
market * my_market
The market I am in.
Definition: scheduler.h:159
size_t my_arena_index
Index of the arena slot the scheduler occupies now, or occupied last time.
Definition: scheduler.h:72
__TBB_atomic intptr_t my_small_task_count
Number of small tasks that have been allocated by this scheduler.
Definition: scheduler.h:383
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
bool outermost
Indicates that a scheduler is on outermost level.
Definition: scheduler.h:57
scheduler_properties my_properties
Definition: scheduler.h:95
task * my_dummy_task
Fake root task created by slave threads.
Definition: scheduler.h:173
task * my_return_list
List of small tasks that have been returned to this scheduler by other schedulers.
Definition: scheduler.h:387
static bool is_set(generic_scheduler *s)
Used to check validity of the local scheduler TLS contents.
Definition: governor.cpp:124
task & allocate_task(size_t number_of_bytes, __TBB_CONTEXT_ARG(task *parent, task_group_context *context))
Allocate task object, either from the heap or a free list.
Definition: scheduler.cpp:304
internal::task_prefix & prefix(internal::version_tag *=NULL) const
Get reference to corresponding task_prefix.
Definition: task.h:941
long my_ref_count
Reference count for scheduler.
Definition: scheduler.h:177

References __TBB_ASSERT, __TBB_CONTEXT_ARG, allocate_task(), tbb::internal::governor::is_set(), ITT_SYNC_CREATE, tbb::internal::scheduler_state::my_arena_index, my_dummy_task, tbb::internal::scheduler_state::my_innermost_running_task, tbb::internal::scheduler_state::my_properties, my_return_list, tbb::internal::scheduler_properties::outermost, and tbb::task::prefix().

Here is the call graph for this function:

Member Function Documentation

◆ acquire_task_pool()

void tbb::internal::generic_scheduler::acquire_task_pool ( ) const
inline

Locks the local task pool.

Garbles my_arena_slot->task_pool for the duration of the lock. Requires correctly set my_arena_slot->task_pool_ptr.

ATTENTION: This method is mostly the same as generic_scheduler::lock_task_pool(), with a little different logic of slot state checks (slot is either locked or points to our task pool). Thus if either of them is changed, consider changing the counterpart as well.

Definition at line 460 of file scheduler.cpp.

460  {
461  if ( !is_task_pool_published() )
462  return; // we are not in arena - nothing to lock
463  bool sync_prepare_done = false;
464  for( atomic_backoff b;;b.pause() ) {
465 #if TBB_USE_ASSERT
466  __TBB_ASSERT( my_arena_slot == my_arena->my_slots + my_arena_index, "invalid arena slot index" );
467  // Local copy of the arena slot task pool pointer is necessary for the next
468  // assertion to work correctly to exclude asynchronous state transition effect.
469  task** tp = my_arena_slot->task_pool;
470  __TBB_ASSERT( tp == LockedTaskPool || tp == my_arena_slot->task_pool_ptr, "slot ownership corrupt?" );
471 #endif
474  {
475  // We acquired our own slot
476  ITT_NOTIFY(sync_acquired, my_arena_slot);
477  break;
478  }
479  else if( !sync_prepare_done ) {
480  // Start waiting
481  ITT_NOTIFY(sync_prepare, my_arena_slot);
482  sync_prepare_done = true;
483  }
484  // Someone else acquired a lock, so pause and do exponential backoff.
485  }
486  __TBB_ASSERT( my_arena_slot->task_pool == LockedTaskPool, "not really acquired task pool" );
487 } // generic_scheduler::acquire_task_pool
#define LockedTaskPool
Definition: scheduler.h:47
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
arena_slot my_slots[1]
Definition: arena.h:300
size_t my_arena_index
Index of the arena slot the scheduler occupies now, or occupied last time.
Definition: scheduler.h:72
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
atomic< T > & as_atomic(T &t)
Definition: atomic.h:547
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, tbb::internal::as_atomic(), is_task_pool_published(), ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena::my_slots, tbb::internal::atomic_backoff::pause(), tbb::internal::arena_slot_line1::task_pool, and tbb::internal::arena_slot_line2::task_pool_ptr.

Referenced by cleanup_master(), get_task(), and prepare_task_pool().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ allocate_task()

task & tbb::internal::generic_scheduler::allocate_task ( size_t  number_of_bytes,
__TBB_CONTEXT_ARG(task *parent, task_group_context *context)   
)

Allocate task object, either from the heap or a free list.

Returns uninitialized task object with initialized prefix.

Definition at line 304 of file scheduler.cpp.

305  {
306  GATHER_STATISTIC(++my_counters.active_tasks);
307  task *t;
308  if( number_of_bytes<=quick_task_size ) {
309 #if __TBB_HOARD_NONLOCAL_TASKS
310  if( (t = my_nonlocal_free_list) ) {
311  GATHER_STATISTIC(--my_counters.free_list_length);
312  __TBB_ASSERT( t->state()==task::freed, "free list of tasks is corrupted" );
313  my_nonlocal_free_list = t->prefix().next;
314  } else
315 #endif
316  if( (t = my_free_list) ) {
317  GATHER_STATISTIC(--my_counters.free_list_length);
318  __TBB_ASSERT( t->state()==task::freed, "free list of tasks is corrupted" );
319  my_free_list = t->prefix().next;
320  } else if( my_return_list ) {
321  // No fence required for read of my_return_list above, because __TBB_FetchAndStoreW has a fence.
322  t = (task*)__TBB_FetchAndStoreW( &my_return_list, 0 ); // with acquire
323  __TBB_ASSERT( t, "another thread emptied the my_return_list" );
324  __TBB_ASSERT( t->prefix().origin==this, "task returned to wrong my_return_list" );
325  ITT_NOTIFY( sync_acquired, &my_return_list );
326  my_free_list = t->prefix().next;
327  } else {
329 #if __TBB_COUNT_TASK_NODES
330  ++my_task_node_count;
331 #endif /* __TBB_COUNT_TASK_NODES */
332  t->prefix().origin = this;
333  t->prefix().next = 0;
335  }
336 #if __TBB_PREFETCHING
337  task *t_next = t->prefix().next;
338  if( !t_next ) { // the task was last in the list
339 #if __TBB_HOARD_NONLOCAL_TASKS
340  if( my_free_list )
341  t_next = my_free_list;
342  else
343 #endif
344  if( my_return_list ) // enable prefetching, gives speedup
345  t_next = my_free_list = (task*)__TBB_FetchAndStoreW( &my_return_list, 0 );
346  }
347  if( t_next ) { // gives speedup for both cache lines
348  __TBB_cl_prefetch(t_next);
349  __TBB_cl_prefetch(&t_next->prefix());
350  }
351 #endif /* __TBB_PREFETCHING */
352  } else {
353  GATHER_STATISTIC(++my_counters.big_tasks);
354  t = (task*)((char*)NFS_Allocate( 1, task_prefix_reservation_size+number_of_bytes, NULL ) + task_prefix_reservation_size );
355 #if __TBB_COUNT_TASK_NODES
356  ++my_task_node_count;
357 #endif /* __TBB_COUNT_TASK_NODES */
358  t->prefix().origin = NULL;
359  }
360  task_prefix& p = t->prefix();
361 #if __TBB_TASK_GROUP_CONTEXT
362  p.context = context;
363 #endif /* __TBB_TASK_GROUP_CONTEXT */
364  // Obsolete. But still in use, so has to be assigned correct value here.
365  p.owner = this;
366  p.ref_count = 0;
367  // Obsolete. Assign some not outrageously out-of-place value for a while.
368  p.depth = 0;
369  p.parent = parent;
370  // In TBB 2.1 and later, the constructor for task sets extra_state to indicate the version of the tbb/task.h header.
371  // In TBB 2.0 and earlier, the constructor leaves extra_state as zero.
372  p.extra_state = 0;
373  p.affinity = 0;
374  p.state = task::allocated;
375  __TBB_ISOLATION_EXPR( p.isolation = no_isolation );
376  return *t;
377 }
void *__TBB_EXPORTED_FUNC NFS_Allocate(size_t n_element, size_t element_size, void *hint)
Allocate memory on cache/sector line boundary.
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
const isolation_tag no_isolation
Definition: task.h:129
task object is freshly allocated or recycled.
Definition: task.h:620
#define __TBB_ISOLATION_EXPR(isolation)
void const char const char int ITT_FORMAT __itt_group_sync p
#define __TBB_cl_prefetch(p)
Definition: mic_common.h:37
__TBB_atomic intptr_t my_small_task_count
Number of small tasks that have been allocated by this scheduler.
Definition: scheduler.h:383
task * my_free_list
Free list of small tasks that can be reused.
Definition: scheduler.h:165
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
#define GATHER_STATISTIC(x)
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id parent
task object is on free list, or is going to be put there, or was just taken off.
Definition: task.h:622
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
task * my_return_list
List of small tasks that have been returned to this scheduler by other schedulers.
Definition: scheduler.h:387
static const size_t quick_task_size
If sizeof(task) is <=quick_task_size, it is handled on a free list instead of malloc'd.
Definition: scheduler.h:131
internal::task_prefix & prefix(internal::version_tag *=NULL) const
Get reference to corresponding task_prefix.
Definition: task.h:941
const size_t task_prefix_reservation_size
Number of bytes reserved for a task prefix.

References __TBB_ASSERT, __TBB_cl_prefetch, __TBB_ISOLATION_EXPR, tbb::task::allocated, tbb::task::freed, GATHER_STATISTIC, ITT_NOTIFY, my_free_list, my_return_list, my_small_task_count, tbb::internal::NFS_Allocate(), tbb::internal::no_isolation, p, parent, tbb::task::prefix(), quick_task_size, tbb::task::state(), and tbb::internal::task_prefix_reservation_size.

Referenced by generic_scheduler(), and prepare_for_spawning().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ assert_task_pool_valid()

void tbb::internal::generic_scheduler::assert_task_pool_valid ( ) const
inline

Definition at line 323 of file scheduler.h.

323 {}

Referenced by local_spawn(), prepare_task_pool(), and tbb::task::self().

Here is the caller graph for this function:

◆ attach_arena()

void tbb::internal::generic_scheduler::attach_arena ( arena a,
size_t  index,
bool  is_master 
)

Definition at line 40 of file arena.cpp.

40  {
41  __TBB_ASSERT( a->my_market == my_market, NULL );
42  my_arena = a;
43  my_arena_index = index;
44  my_arena_slot = a->my_slots + index;
45  attach_mailbox( affinity_id(index+1) );
46  if ( is_master && my_inbox.is_idle_state( true ) ) {
47  // Master enters an arena with its own task to be executed. It means that master is not
48  // going to enter stealing loop and take affinity tasks.
49  my_inbox.set_is_idle( false );
50  }
51 #if __TBB_TASK_GROUP_CONTEXT
52  // Context to be used by root tasks by default (if the user has not specified one).
53  if( !is_master )
54  my_dummy_task->prefix().context = a->my_default_ctx;
55 #endif /* __TBB_TASK_GROUP_CONTEXT */
56 #if __TBB_TASK_PRIORITY
57  // In the current implementation master threads continue processing even when
58  // there are other masters with higher priority. Only TBB worker threads are
59  // redistributed between arenas based on the latters' priority. Thus master
60  // threads use arena's top priority as a reference point (in contrast to workers
61  // that use my_market->my_global_top_priority).
62  if( is_master ) {
63  my_ref_top_priority = &a->my_top_priority;
64  my_ref_reload_epoch = &a->my_reload_epoch;
65  }
66  my_local_reload_epoch = *my_ref_reload_epoch;
67  __TBB_ASSERT( !my_offloaded_tasks, NULL );
68 #endif /* __TBB_TASK_PRIORITY */
69 }
unsigned short affinity_id
An id as used for specifying affinity.
Definition: task.h:124
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
market * my_market
The market I am in.
Definition: scheduler.h:159
size_t my_arena_index
Index of the arena slot the scheduler occupies now, or occupied last time.
Definition: scheduler.h:72
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void attach_mailbox(affinity_id id)
Definition: scheduler.h:589
task * my_dummy_task
Fake root task created by slave threads.
Definition: scheduler.h:173
void set_is_idle(bool value)
Indicate whether thread that reads this mailbox is idle.
Definition: mailbox.h:215
internal::task_prefix & prefix(internal::version_tag *=NULL) const
Get reference to corresponding task_prefix.
Definition: task.h:941
bool is_idle_state(bool value) const
Indicate whether thread that reads this mailbox is idle.
Definition: mailbox.h:222
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, attach_mailbox(), tbb::internal::mail_inbox::is_idle_state(), tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, my_dummy_task, tbb::internal::scheduler_state::my_inbox, tbb::internal::arena_base::my_market, my_market, tbb::internal::arena::my_slots, tbb::task::prefix(), and tbb::internal::mail_inbox::set_is_idle().

Referenced by nested_arena_entry().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ attach_mailbox()

void tbb::internal::generic_scheduler::attach_mailbox ( affinity_id  id)
inline

Definition at line 589 of file scheduler.h.

589  {
590  __TBB_ASSERT(id>0,NULL);
592  my_affinity_id = id;
593 }
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id id
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
void attach(mail_outbox &putter)
Attach inbox to a corresponding outbox.
Definition: mailbox.h:197
mail_outbox & mailbox(affinity_id id)
Get reference to mailbox corresponding to given affinity_id.
Definition: arena.h:208
affinity_id my_affinity_id
The mailbox id assigned to this scheduler.
Definition: scheduler.h:93
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, tbb::internal::mail_inbox::attach(), id, tbb::internal::arena::mailbox(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, and tbb::internal::scheduler_state::my_inbox.

Referenced by attach_arena().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ can_steal()

bool tbb::internal::generic_scheduler::can_steal ( )
inline

Returns true if stealing is allowed.

Definition at line 195 of file scheduler.h.

195  {
196  int anchor;
197  // TODO IDEA: Add performance warning?
198 #if __TBB_ipf
199  return my_stealing_threshold < (uintptr_t)&anchor && (uintptr_t)__TBB_get_bsp() < my_rsb_stealing_threshold;
200 #else
201  return my_stealing_threshold < (uintptr_t)&anchor;
202 #endif
203  }
void * __TBB_get_bsp()
Retrieves the current RSE backing store pointer. IA64 specific.
uintptr_t my_stealing_threshold
Position in the call stack specifying its maximal filling when stealing is still allowed.
Definition: scheduler.h:142

References __TBB_get_bsp(), and my_stealing_threshold.

Here is the call graph for this function:

◆ cleanup_master()

bool tbb::internal::generic_scheduler::cleanup_master ( bool  blocking_terminate)

Perform necessary cleanup when a master thread stops using TBB.

Definition at line 1304 of file scheduler.cpp.

1304  {
1305  arena* const a = my_arena;
1306  market * const m = my_market;
1307  __TBB_ASSERT( my_market, NULL );
1308  if( a && is_task_pool_published() ) {
1312  {
1313  // Local task pool is empty
1314  leave_task_pool();
1315  }
1316  else {
1317  // Master's local task pool may e.g. contain proxies of affinitized tasks.
1319  __TBB_ASSERT ( governor::is_set(this), "TLS slot is cleared before the task pool cleanup" );
1322  __TBB_ASSERT ( governor::is_set(this), "Other thread reused our TLS key during the task pool cleanup" );
1323  }
1324  }
1325 #if __TBB_ARENA_OBSERVER
1326  if( a )
1327  a->my_observers.notify_exit_observers( my_last_local_observer, /*worker=*/false );
1328 #endif
1329 #if __TBB_SCHEDULER_OBSERVER
1330  the_global_observer_list.notify_exit_observers( my_last_global_observer, /*worker=*/false );
1331 #endif /* __TBB_SCHEDULER_OBSERVER */
1332 #if _WIN32||_WIN64
1333  m->unregister_master( master_exec_resource );
1334 #endif /* _WIN32||_WIN64 */
1335  if( a ) {
1336  __TBB_ASSERT(a->my_slots+0 == my_arena_slot, NULL);
1337 #if __TBB_STATISTICS
1338  *my_arena_slot->my_counters += my_counters;
1339 #endif /* __TBB_STATISTICS */
1341  }
1342 #if __TBB_TASK_GROUP_CONTEXT
1343  else { // task_group_context ownership was not transferred to arena
1344  default_context()->~task_group_context();
1345  NFS_Free(default_context());
1346  }
1347  context_state_propagation_mutex_type::scoped_lock lock(the_context_state_propagation_mutex);
1348  my_market->my_masters.remove( *this );
1349  lock.release();
1350 #endif /* __TBB_TASK_GROUP_CONTEXT */
1351  my_arena_slot = NULL; // detached from slot
1352  free_scheduler(); // do not use scheduler state after this point
1353 
1354  if( a )
1355  a->on_thread_leaving<arena::ref_external>();
1356  // If there was an associated arena, it added a public market reference
1357  return m->release( /*is_public*/ a != NULL, blocking_terminate );
1358 }
#define EmptyTaskPool
Definition: scheduler.h:46
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
__TBB_atomic size_t head
Index of the first ready task in the deque.
void release_task_pool() const
Unlocks the local task pool.
Definition: scheduler.cpp:489
static const unsigned ref_external
Reference increment values for externals and workers.
Definition: arena.h:230
generic_scheduler * my_scheduler
Scheduler of the thread attached to the slot.
market * my_market
The market I am in.
Definition: scheduler.h:159
void free_scheduler()
Destroy and deallocate this scheduler object.
Definition: scheduler.cpp:264
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void * lock
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
void acquire_task_pool() const
Locks the local task pool.
Definition: scheduler.cpp:460
task * my_dummy_task
Fake root task created by slave threads.
Definition: scheduler.h:173
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:739
static bool is_set(generic_scheduler *s)
Used to check validity of the local scheduler TLS contents.
Definition: governor.cpp:124
void __TBB_EXPORTED_FUNC NFS_Free(void *)
Free memory allocated by NFS_Allocate.
virtual void local_wait_for_all(task &parent, task *child)=0
void leave_task_pool()
Leave the task pool.
Definition: scheduler.cpp:1226
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:717
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_with_release(), acquire_task_pool(), EmptyTaskPool, free_scheduler(), tbb::internal::arena_slot_line1::head, tbb::internal::governor::is_set(), is_task_pool_published(), leave_task_pool(), local_wait_for_all(), lock, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_slot, my_dummy_task, my_market, tbb::internal::arena_slot_line1::my_scheduler, tbb::internal::arena::my_slots, tbb::internal::NFS_Free(), tbb::internal::arena::on_thread_leaving(), tbb::internal::arena::ref_external, tbb::internal::market::release(), release_task_pool(), tbb::internal::arena_slot_line2::tail, and tbb::internal::arena_slot_line1::task_pool.

Here is the call graph for this function:

◆ cleanup_worker()

void tbb::internal::generic_scheduler::cleanup_worker ( void arg,
bool  worker 
)
static

Perform necessary cleanup when a worker thread finishes.

Definition at line 1294 of file scheduler.cpp.

1294  {
1296  __TBB_ASSERT( !s.my_arena_slot, "cleaning up attached worker" );
1297 #if __TBB_SCHEDULER_OBSERVER
1298  if ( worker ) // can be called by master for worker, do not notify master twice
1299  the_global_observer_list.notify_exit_observers( s.my_last_global_observer, /*worker=*/true );
1300 #endif /* __TBB_SCHEDULER_OBSERVER */
1301  s.free_scheduler();
1302 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
void const char const char int ITT_FORMAT __itt_group_sync s

References __TBB_ASSERT, and s.

Referenced by tbb::internal::market::cleanup().

Here is the caller graph for this function:

◆ commit_relocated_tasks()

void tbb::internal::generic_scheduler::commit_relocated_tasks ( size_t  new_tail)
inline

Makes relocated tasks visible to thieves and releases the local task pool.

Obviously, the task pool must be locked when calling this method.

Definition at line 641 of file scheduler.h.

641  {
643  "Task pool must be locked when calling commit_relocated_tasks()" );
645  // Tail is updated last to minimize probability of a thread making arena
646  // snapshot being misguided into thinking that this task pool is empty.
647  __TBB_store_release( my_arena_slot->tail, new_tail );
649 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
__TBB_atomic size_t head
Index of the first ready task in the deque.
void release_task_pool() const
Unlocks the local task pool.
Definition: scheduler.cpp:489
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void __TBB_store_relaxed(volatile T &location, V value)
Definition: tbb_machine.h:743
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
bool is_local_task_pool_quiescent() const
Definition: scheduler.h:555
#define __TBB_store_release
Definition: tbb_machine.h:861

References __TBB_ASSERT, tbb::internal::__TBB_store_relaxed(), __TBB_store_release, tbb::internal::arena_slot_line1::head, is_local_task_pool_quiescent(), tbb::internal::scheduler_state::my_arena_slot, release_task_pool(), and tbb::internal::arena_slot_line2::tail.

Referenced by prepare_task_pool().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ commit_spawned_tasks()

void tbb::internal::generic_scheduler::commit_spawned_tasks ( size_t  new_tail)
inline

Makes newly spawned tasks visible to thieves.

Definition at line 632 of file scheduler.h.

632  {
633  __TBB_ASSERT ( new_tail <= my_arena_slot->my_task_pool_size, "task deque end was overwritten" );
634  // emit "task was released" signal
635  ITT_NOTIFY(sync_releasing, (void*)((uintptr_t)my_arena_slot+sizeof(uintptr_t)));
636  // Release fence is necessary to make sure that previously stored task pointers
637  // are visible to thieves.
639 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:717

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), ITT_NOTIFY, tbb::internal::scheduler_state::my_arena_slot, sync_releasing, and tbb::internal::arena_slot_line2::tail.

Referenced by local_spawn().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ create_master()

generic_scheduler * tbb::internal::generic_scheduler::create_master ( arena a)
static

Initialize a scheduler for a master thread.

Definition at line 1252 of file scheduler.cpp.

1252  {
1253  // add an internal market reference; the public reference is possibly added in create_arena
1254  generic_scheduler* s = allocate_scheduler( market::global_market(/*is_public=*/false) );
1255  __TBB_ASSERT( !s->my_arena, NULL );
1256  __TBB_ASSERT( s->my_market, NULL );
1257  task& t = *s->my_dummy_task;
1258  s->my_properties.type = scheduler_properties::master;
1259  t.prefix().ref_count = 1;
1260 #if __TBB_TASK_GROUP_CONTEXT
1261  t.prefix().context = new ( NFS_Allocate(1, sizeof(task_group_context), NULL) )
1263 #if __TBB_FP_CONTEXT
1264  s->default_context()->capture_fp_settings();
1265 #endif
1266  // Do not call init_stack_info before the scheduler is set as master or worker.
1267  s->init_stack_info();
1268  context_state_propagation_mutex_type::scoped_lock lock(the_context_state_propagation_mutex);
1269  s->my_market->my_masters.push_front( *s );
1270  lock.release();
1271 #endif /* __TBB_TASK_GROUP_CONTEXT */
1272  if( a ) {
1273  // Master thread always occupies the first slot
1274  s->attach_arena( a, /*index*/0, /*is_master*/true );
1275  s->my_arena_slot->my_scheduler = s;
1276  a->my_default_ctx = s->default_context(); // also transfers implied ownership
1277  }
1278  __TBB_ASSERT( s->my_arena_index == 0, "Master thread must occupy the first slot in its arena" );
1280 
1281 #if _WIN32||_WIN64
1282  s->my_market->register_master( s->master_exec_resource );
1283 #endif /* _WIN32||_WIN64 */
1284  // Process any existing observers.
1285 #if __TBB_ARENA_OBSERVER
1286  __TBB_ASSERT( !a || a->my_observers.empty(), "Just created arena cannot have any observers associated with it" );
1287 #endif
1288 #if __TBB_SCHEDULER_OBSERVER
1289  the_global_observer_list.notify_entry_observers( s->my_last_global_observer, /*worker=*/false );
1290 #endif /* __TBB_SCHEDULER_OBSERVER */
1291  return s;
1292 }
void *__TBB_EXPORTED_FUNC NFS_Allocate(size_t n_element, size_t element_size, void *hint)
Allocate memory on cache/sector line boundary.
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
static market & global_market(bool is_public, unsigned max_num_workers=0, size_t stack_size=0)
Factory method creating new market object.
Definition: market.cpp:100
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void * lock
static void sign_on(generic_scheduler *s)
Register TBB scheduler instance in thread-local storage.
Definition: governor.cpp:128
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
void const char const char int ITT_FORMAT __itt_group_sync s
generic_scheduler * allocate_scheduler(market &m)
Definition: scheduler.cpp:41

References __TBB_ASSERT, tbb::internal::allocate_scheduler(), tbb::task_group_context::default_traits, tbb::internal::market::global_market(), tbb::task_group_context::isolated, lock, tbb::internal::scheduler_properties::master, tbb::internal::NFS_Allocate(), tbb::task::prefix(), s, and tbb::internal::governor::sign_on().

Referenced by tbb::internal::governor::init_scheduler(), and tbb::internal::governor::init_scheduler_weak().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ create_worker()

generic_scheduler * tbb::internal::generic_scheduler::create_worker ( market m,
size_t  index 
)
static

Initialize a scheduler for a worker thread.

Definition at line 1239 of file scheduler.cpp.

1239  {
1241  __TBB_ASSERT(index, "workers should have index > 0");
1242  s->my_arena_index = index; // index is not a real slot in arena yet
1243  s->my_dummy_task->prefix().ref_count = 2;
1244  s->my_properties.type = scheduler_properties::worker;
1245  // Do not call init_stack_info before the scheduler is set as master or worker.
1246  s->init_stack_info();
1248  return s;
1249 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
static void sign_on(generic_scheduler *s)
Register TBB scheduler instance in thread-local storage.
Definition: governor.cpp:128
void const char const char int ITT_FORMAT __itt_group_sync s
generic_scheduler * allocate_scheduler(market &m)
Definition: scheduler.cpp:41

References __TBB_ASSERT, tbb::internal::allocate_scheduler(), s, tbb::internal::governor::sign_on(), and tbb::internal::scheduler_properties::worker.

Referenced by tbb::internal::market::create_one_job().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ deallocate_task()

void tbb::internal::generic_scheduler::deallocate_task ( task t)
inline

Return task object to the memory allocator.

Definition at line 605 of file scheduler.h.

605  {
606 #if TBB_USE_ASSERT
607  task_prefix& p = t.prefix();
608  p.state = 0xFF;
609  p.extra_state = 0xFF;
610  poison_pointer(p.next);
611 #endif /* TBB_USE_ASSERT */
613 #if __TBB_COUNT_TASK_NODES
614  --my_task_node_count;
615 #endif /* __TBB_COUNT_TASK_NODES */
616 }
void const char const char int ITT_FORMAT __itt_group_sync p
void poison_pointer(T *__TBB_atomic &)
Definition: tbb_stddef.h:309
void __TBB_EXPORTED_FUNC NFS_Free(void *)
Free memory allocated by NFS_Allocate.
const size_t task_prefix_reservation_size
Number of bytes reserved for a task prefix.

References tbb::internal::NFS_Free(), p, tbb::internal::poison_pointer(), tbb::task::prefix(), and tbb::internal::task_prefix_reservation_size.

Referenced by free_nonlocal_small_task(), free_scheduler(), and free_task().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ enqueue()

void tbb::internal::generic_scheduler::enqueue ( task t,
void reserved 
)
virtual

For internal use only.

Implements tbb::internal::scheduler.

Definition at line 714 of file scheduler.cpp.

714  {
716  // these redirections are due to bw-compatibility, consider reworking some day
717  __TBB_ASSERT( s->my_arena, "thread is not in any arena" );
718  s->my_arena->enqueue_task(t, (intptr_t)prio, s->my_random );
719 }
static generic_scheduler * local_scheduler()
Obtain the thread-local instance of the TBB scheduler.
Definition: governor.h:126
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
void const char const char int ITT_FORMAT __itt_group_sync s

References __TBB_ASSERT, tbb::internal::governor::local_scheduler(), and s.

Here is the call graph for this function:

◆ free_nonlocal_small_task()

void tbb::internal::generic_scheduler::free_nonlocal_small_task ( task t)

Free a small task t that that was allocated by a different scheduler.

Definition at line 379 of file scheduler.cpp.

379  {
380  __TBB_ASSERT( t.state()==task::freed, NULL );
381  generic_scheduler& s = *static_cast<generic_scheduler*>(t.prefix().origin);
382  __TBB_ASSERT( &s!=this, NULL );
383  for(;;) {
384  task* old = s.my_return_list;
385  if( old==plugged_return_list() )
386  break;
387  // Atomically insert t at head of s.my_return_list
388  t.prefix().next = old;
389  ITT_NOTIFY( sync_releasing, &s.my_return_list );
390  if( as_atomic(s.my_return_list).compare_and_swap(&t, old )==old ) {
391 #if __TBB_PREFETCHING
392  __TBB_cl_evict(&t.prefix());
393  __TBB_cl_evict(&t);
394 #endif
395  return;
396  }
397  }
398  deallocate_task(t);
399  if( __TBB_FetchAndDecrementWrelease( &s.my_small_task_count )==1 ) {
400  // We freed the last task allocated by scheduler s, so it's our responsibility
401  // to free the scheduler.
402  NFS_Free( &s );
403  }
404 }
#define __TBB_FetchAndDecrementWrelease(P)
Definition: tbb_machine.h:315
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
#define __TBB_cl_evict(p)
Definition: mic_common.h:38
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
void deallocate_task(task &t)
Return task object to the memory allocator.
Definition: scheduler.h:605
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
static task * plugged_return_list()
Special value used to mark my_return_list as not taking any more entries.
Definition: scheduler.h:380
task object is on free list, or is going to be put there, or was just taken off.
Definition: task.h:622
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
void const char const char int ITT_FORMAT __itt_group_sync s
void __TBB_EXPORTED_FUNC NFS_Free(void *)
Free memory allocated by NFS_Allocate.
atomic< T > & as_atomic(T &t)
Definition: atomic.h:547

References __TBB_ASSERT, __TBB_cl_evict, __TBB_FetchAndDecrementWrelease, tbb::internal::as_atomic(), deallocate_task(), tbb::task::freed, ITT_NOTIFY, tbb::internal::NFS_Free(), plugged_return_list(), tbb::task::prefix(), s, tbb::task::state(), and sync_releasing.

Referenced by free_scheduler(), and free_task().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ free_scheduler()

void tbb::internal::generic_scheduler::free_scheduler ( )

Destroy and deallocate this scheduler object.

Definition at line 264 of file scheduler.cpp.

264  {
265  __TBB_ASSERT( !my_arena_slot, NULL );
266 #if __TBB_PREVIEW_CRITICAL_TASKS
267  __TBB_ASSERT( !my_properties.has_taken_critical_task, "Critical tasks miscount." );
268 #endif
269 #if __TBB_TASK_GROUP_CONTEXT
270  cleanup_local_context_list();
271 #endif /* __TBB_TASK_GROUP_CONTEXT */
272  free_task<small_local_task>( *my_dummy_task );
273 
274 #if __TBB_HOARD_NONLOCAL_TASKS
275  while( task* t = my_nonlocal_free_list ) {
276  task_prefix& p = t->prefix();
277  my_nonlocal_free_list = p.next;
278  __TBB_ASSERT( p.origin && p.origin!=this, NULL );
280  }
281 #endif
282  // k accounts for a guard reference and each task that we deallocate.
283  intptr_t k = 1;
284  for(;;) {
285  while( task* t = my_free_list ) {
286  my_free_list = t->prefix().next;
287  deallocate_task(*t);
288  ++k;
289  }
291  break;
292  my_free_list = (task*)__TBB_FetchAndStoreW( &my_return_list, (intptr_t)plugged_return_list() );
293  }
294 #if __TBB_COUNT_TASK_NODES
295  my_market->update_task_node_count( my_task_node_count );
296 #endif /* __TBB_COUNT_TASK_NODES */
297  // Update my_small_task_count last. Doing so sooner might cause another thread to free *this.
298  __TBB_ASSERT( my_small_task_count>=k, "my_small_task_count corrupted" );
299  governor::sign_off(this);
300  if( __TBB_FetchAndAddW( &my_small_task_count, -k )==k )
301  NFS_Free( this );
302 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
static void sign_off(generic_scheduler *s)
Unregister TBB scheduler instance from thread-local storage.
Definition: governor.cpp:149
market * my_market
The market I am in.
Definition: scheduler.h:159
void const char const char int ITT_FORMAT __itt_group_sync p
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
__TBB_atomic intptr_t my_small_task_count
Number of small tasks that have been allocated by this scheduler.
Definition: scheduler.h:383
task * my_free_list
Free list of small tasks that can be reused.
Definition: scheduler.h:165
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
void deallocate_task(task &t)
Return task object to the memory allocator.
Definition: scheduler.h:605
static task * plugged_return_list()
Special value used to mark my_return_list as not taking any more entries.
Definition: scheduler.h:380
scheduler_properties my_properties
Definition: scheduler.h:95
task * my_dummy_task
Fake root task created by slave threads.
Definition: scheduler.h:173
task * my_return_list
List of small tasks that have been returned to this scheduler by other schedulers.
Definition: scheduler.h:387
void __TBB_EXPORTED_FUNC NFS_Free(void *)
Free memory allocated by NFS_Allocate.
internal::task_prefix & prefix(internal::version_tag *=NULL) const
Get reference to corresponding task_prefix.
Definition: task.h:941
void free_nonlocal_small_task(task &t)
Free a small task t that that was allocated by a different scheduler.
Definition: scheduler.cpp:379

References __TBB_ASSERT, deallocate_task(), free_nonlocal_small_task(), tbb::internal::scheduler_state::my_arena_slot, my_dummy_task, my_free_list, my_market, tbb::internal::scheduler_state::my_properties, my_return_list, my_small_task_count, tbb::internal::NFS_Free(), p, plugged_return_list(), tbb::task::prefix(), and tbb::internal::governor::sign_off().

Referenced by cleanup_master().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ free_task()

template<free_task_hint hint>
void tbb::internal::generic_scheduler::free_task ( task t)

Put task on free list.

Does not call destructor.

Definition at line 652 of file scheduler.h.

652  {
653 #if __TBB_HOARD_NONLOCAL_TASKS
654  static const int h = hint&(~local_task);
655 #else
656  static const free_task_hint h = hint;
657 #endif
658  GATHER_STATISTIC(--my_counters.active_tasks);
659  task_prefix& p = t.prefix();
660  // Verify that optimization hints are correct.
661  __TBB_ASSERT( h!=small_local_task || p.origin==this, NULL );
662  __TBB_ASSERT( !(h&small_task) || p.origin, NULL );
663  __TBB_ASSERT( !(h&local_task) || (!p.origin || uintptr_t(p.origin) > uintptr_t(4096)), "local_task means allocated");
664  poison_value(p.depth);
665  poison_value(p.ref_count);
666  poison_pointer(p.owner);
667  __TBB_ASSERT( 1L<<t.state() & (1L<<task::executing|1L<<task::allocated), NULL );
668  p.state = task::freed;
669  if( h==small_local_task || p.origin==this ) {
670  GATHER_STATISTIC(++my_counters.free_list_length);
671  p.next = my_free_list;
672  my_free_list = &t;
673  } else if( !(h&local_task) && p.origin && uintptr_t(p.origin) < uintptr_t(4096) ) {
674  // a special value reserved for future use, do nothing since
675  // origin is not pointing to a scheduler instance
676  } else if( !(h&local_task) && p.origin ) {
677  GATHER_STATISTIC(++my_counters.free_list_length);
678 #if __TBB_HOARD_NONLOCAL_TASKS
679  if( !(h&no_cache) ) {
680  p.next = my_nonlocal_free_list;
681  my_nonlocal_free_list = &t;
682  } else
683 #endif
685  } else {
686  GATHER_STATISTIC(--my_counters.big_tasks);
687  deallocate_task(t);
688  }
689 }
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function h
Disable caching for a small task.
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
Task is known to have been allocated by this scheduler.
task object is freshly allocated or recycled.
Definition: task.h:620
Task is known to be a small task.
task is running, and will be destroyed after method execute() completes.
Definition: task.h:614
void const char const char int ITT_FORMAT __itt_group_sync p
free_task_hint
Optimization hint to free_task that enables it omit unnecessary tests and code.
#define poison_value(g)
task * my_free_list
Free list of small tasks that can be reused.
Definition: scheduler.h:165
void deallocate_task(task &t)
Return task object to the memory allocator.
Definition: scheduler.h:605
#define GATHER_STATISTIC(x)
task object is on free list, or is going to be put there, or was just taken off.
Definition: task.h:622
void poison_pointer(T *__TBB_atomic &)
Definition: tbb_stddef.h:309
Bitwise-OR of local_task and small_task.
void free_nonlocal_small_task(task &t)
Free a small task t that that was allocated by a different scheduler.
Definition: scheduler.cpp:379

References __TBB_ASSERT, tbb::task::allocated, deallocate_task(), tbb::task::executing, free_nonlocal_small_task(), tbb::task::freed, GATHER_STATISTIC, h, tbb::internal::local_task, my_free_list, tbb::internal::no_cache, p, tbb::internal::poison_pointer(), poison_value, tbb::task::prefix(), tbb::internal::small_local_task, tbb::internal::small_task, and tbb::task::state().

Referenced by tbb::interface5::internal::task_base::destroy(), tbb::internal::allocate_additional_child_of_proxy::free(), tbb::internal::allocate_continuation_proxy::free(), tbb::internal::allocate_child_proxy::free(), and tbb::internal::auto_empty_task::~auto_empty_task().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ get_mailbox_task()

task * tbb::internal::generic_scheduler::get_mailbox_task ( __TBB_ISOLATION_EXPR(isolation_tag isolation)  )

Attempt to get a task from the mailbox.

Gets a task only if it has not been executed by its sender or a thief that has stolen it from the sender's task pool. Otherwise returns NULL.

This method is intended to be used only by the thread extracting the proxy from its mailbox. (In contrast to local task pool, mailbox can be read only by its owner).

Definition at line 1200 of file scheduler.cpp.

1200  {
1201  __TBB_ASSERT( my_affinity_id>0, "not in arena" );
1202  while ( task_proxy* const tp = my_inbox.pop( __TBB_ISOLATION_EXPR( isolation ) ) ) {
1203  if ( task* result = tp->extract_task<task_proxy::mailbox_bit>() ) {
1204  ITT_NOTIFY( sync_acquired, my_inbox.outbox() );
1205  result->prefix().extra_state |= es_task_is_stolen;
1206  return result;
1207  }
1208  // We have exclusive access to the proxy, and can destroy it.
1209  free_task<no_cache_small_task>(*tp);
1210  }
1211  return NULL;
1212 }
static const intptr_t mailbox_bit
Definition: mailbox.h:35
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
#define __TBB_ISOLATION_EXPR(isolation)
Set if the task has been stolen.
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
task_proxy * pop(__TBB_ISOLATION_EXPR(isolation_tag isolation))
Get next piece of mail, or NULL if mailbox is empty.
Definition: mailbox.h:206
affinity_id my_affinity_id
The mailbox id assigned to this scheduler.
Definition: scheduler.h:93

References __TBB_ASSERT, __TBB_ISOLATION_EXPR, tbb::internal::es_task_is_stolen, ITT_NOTIFY, tbb::internal::task_proxy::mailbox_bit, tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_inbox, and tbb::internal::mail_inbox::pop().

Here is the call graph for this function:

◆ get_task() [1/2]

task * tbb::internal::generic_scheduler::get_task ( __TBB_ISOLATION_EXPR(isolation_tag isolation)  )
inline

Get a task from the local pool.

Called only by the pool owner. Returns the pointer to the task or NULL if a suitable task is not found. Resets the pool if it is empty.

Definition at line 978 of file scheduler.cpp.

978  {
980  // The current task position in the task pool.
981  size_t T0 = __TBB_load_relaxed( my_arena_slot->tail );
982  // The bounds of available tasks in the task pool. H0 is only used when the head bound is reached.
983  size_t H0 = (size_t)-1, T = T0;
984  task* result = NULL;
985  bool task_pool_empty = false;
986  __TBB_ISOLATION_EXPR( bool tasks_omitted = false );
987  do {
988  __TBB_ASSERT( !result, NULL );
990  atomic_fence();
991  if ( (intptr_t)__TBB_load_relaxed( my_arena_slot->head ) > (intptr_t)T ) {
994  if ( (intptr_t)H0 > (intptr_t)T ) {
995  // The thief has not backed off - nothing to grab.
998  && H0 == T + 1, "victim/thief arbitration algorithm failure" );
1000  // No tasks in the task pool.
1001  task_pool_empty = true;
1002  break;
1003  } else if ( H0 == T ) {
1004  // There is only one task in the task pool.
1006  task_pool_empty = true;
1007  } else {
1008  // Release task pool if there are still some tasks.
1009  // After the release, the tail will be less than T, thus a thief
1010  // will not attempt to get a task at position T.
1012  }
1013  }
1014  __TBB_control_consistency_helper(); // on my_arena_slot->head
1015 #if __TBB_TASK_ISOLATION
1016  result = get_task( T, isolation, tasks_omitted );
1017  if ( result ) {
1019  break;
1020  } else if ( !tasks_omitted ) {
1022  __TBB_ASSERT( T0 == T+1, NULL );
1023  T0 = T;
1024  }
1025 #else
1026  result = get_task( T );
1027 #endif /* __TBB_TASK_ISOLATION */
1028  } while ( !result && !task_pool_empty );
1029 
1030 #if __TBB_TASK_ISOLATION
1031  if ( tasks_omitted ) {
1032  if ( task_pool_empty ) {
1033  // All tasks have been checked. The task pool should be in reset state.
1034  // We just restore the bounds for the available tasks.
1035  // TODO: Does it have sense to move them to the beginning of the task pool?
1037  if ( result ) {
1038  // If we have a task, it should be at H0 position.
1039  __TBB_ASSERT( H0 == T, NULL );
1040  ++H0;
1041  }
1042  __TBB_ASSERT( H0 <= T0, NULL );
1043  if ( H0 < T0 ) {
1044  // Restore the task pool if there are some tasks.
1047  // The release fence is used in publish_task_pool.
1049  // Synchronize with snapshot as we published some tasks.
1051  }
1052  } else {
1053  // A task has been obtained. We need to make a hole in position T.
1055  __TBB_ASSERT( result, NULL );
1056  my_arena_slot->task_pool_ptr[T] = NULL;
1058  // Synchronize with snapshot as we published some tasks.
1059  // TODO: consider some approach not to call wakeup for each time. E.g. check if the tail reached the head.
1061  }
1062 
1063  // Now it is safe to call note_affinity because the task pool is restored.
1064  if ( my_innermost_running_task == result ) {
1065  assert_task_valid( result );
1066  result->note_affinity( my_affinity_id );
1067  }
1068  }
1069 #endif /* __TBB_TASK_ISOLATION */
1070  __TBB_ASSERT( (intptr_t)__TBB_load_relaxed( my_arena_slot->tail ) >= 0, NULL );
1071  __TBB_ASSERT( result || __TBB_ISOLATION_EXPR( tasks_omitted || ) is_quiescent_local_task_pool_reset(), NULL );
1072  return result;
1073 } // generic_scheduler::get_task
void publish_task_pool()
Used by workers to enter the task pool.
Definition: scheduler.cpp:1214
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
#define __TBB_ISOLATION_EXPR(isolation)
bool is_quiescent_local_task_pool_reset() const
Definition: scheduler.h:566
__TBB_atomic size_t head
Index of the first ready task in the deque.
void release_task_pool() const
Unlocks the local task pool.
Definition: scheduler.cpp:489
task * my_innermost_running_task
Innermost task whose task::execute() is running. A dummy task on the outermost level.
Definition: scheduler.h:81
task * get_task(__TBB_ISOLATION_EXPR(isolation_tag isolation))
Get a task from the local pool.
Definition: scheduler.cpp:978
void atomic_fence()
Sequentially consistent full memory fence.
Definition: tbb_machine.h:343
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void __TBB_store_relaxed(volatile T &location, V value)
Definition: tbb_machine.h:743
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
void acquire_task_pool() const
Locks the local task pool.
Definition: scheduler.cpp:460
void poison_pointer(T *__TBB_atomic &)
Definition: tbb_stddef.h:309
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:739
void assert_task_valid(const task *)
void reset_task_pool_and_leave()
Resets head and tail indices to 0, and leaves task pool.
Definition: scheduler.h:624
#define __TBB_control_consistency_helper()
Definition: gcc_generic.h:64
void advertise_new_work()
If necessary, raise a flag that there is new job in arena.
Definition: arena.h:393
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:717
affinity_id my_affinity_id
The mailbox id assigned to this scheduler.
Definition: scheduler.h:93
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, __TBB_control_consistency_helper, __TBB_ISOLATION_EXPR, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_relaxed(), tbb::internal::__TBB_store_with_release(), acquire_task_pool(), tbb::internal::arena::advertise_new_work(), tbb::internal::assert_task_valid(), tbb::atomic_fence(), tbb::internal::arena_slot_line1::head, is_quiescent_local_task_pool_reset(), is_task_pool_published(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::scheduler_state::my_innermost_running_task, tbb::task::note_affinity(), tbb::internal::poison_pointer(), publish_task_pool(), release_task_pool(), reset_task_pool_and_leave(), tbb::internal::arena_slot_line2::tail, tbb::internal::arena_slot_line2::task_pool_ptr, and tbb::internal::arena::wakeup.

Here is the call graph for this function:

◆ get_task() [2/2]

task * tbb::internal::generic_scheduler::get_task ( size_t  T)
inline

Get a task from the local pool at specified location T.

Returns the pointer to the task or NULL if the task cannot be executed, e.g. proxy has been deallocated or isolation constraint is not met. tasks_omitted tells if some tasks have been omitted. Called only by the pool owner. The caller should guarantee that the position T is not available for a thief.

Definition at line 926 of file scheduler.cpp.

928 {
930  || is_local_task_pool_quiescent(), "Is it safe to get a task at position T?" );
931 
932  task* result = my_arena_slot->task_pool_ptr[T];
933  __TBB_ASSERT( !is_poisoned( result ), "The poisoned task is going to be processed" );
934 #if __TBB_TASK_ISOLATION
935  if ( !result )
936  return NULL;
937 
938  bool omit = isolation != no_isolation && isolation != result->prefix().isolation;
939  if ( !omit && !is_proxy( *result ) )
940  return result;
941  else if ( omit ) {
942  tasks_omitted = true;
943  return NULL;
944  }
945 #else
947  if ( !result || !is_proxy( *result ) )
948  return result;
949 #endif /* __TBB_TASK_ISOLATION */
950 
951  task_proxy& tp = static_cast<task_proxy&>(*result);
952  if ( task *t = tp.extract_task<task_proxy::pool_bit>() ) {
953  GATHER_STATISTIC( ++my_counters.proxies_executed );
954  // Following assertion should be true because TBB 2.0 tasks never specify affinity, and hence are not proxied.
955  __TBB_ASSERT( is_version_3_task( *t ), "backwards compatibility with TBB 2.0 broken" );
957  my_innermost_running_task = t; // prepare for calling note_affinity()
958 #if __TBB_TASK_ISOLATION
959  // Task affinity has changed. Postpone calling note_affinity because the task pool is in invalid state.
960  if ( !tasks_omitted )
961 #endif /* __TBB_TASK_ISOLATION */
962  {
964  t->note_affinity( my_affinity_id );
965  }
966  return t;
967  }
968 
969  // Proxy was empty, so it's our responsibility to free it
970  free_task<small_task>( tp );
971 #if __TBB_TASK_ISOLATION
972  if ( tasks_omitted )
973  my_arena_slot->task_pool_ptr[T] = NULL;
974 #endif /* __TBB_TASK_ISOLATION */
975  return NULL;
976 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
const isolation_tag no_isolation
Definition: task.h:129
task * my_innermost_running_task
Innermost task whose task::execute() is running. A dummy task on the outermost level.
Definition: scheduler.h:81
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
static bool is_version_3_task(task &t)
Definition: scheduler.h:133
#define GATHER_STATISTIC(x)
bool is_local_task_pool_quiescent() const
Definition: scheduler.h:555
static bool is_proxy(const task &t)
True if t is a task_proxy.
Definition: scheduler.h:273
void poison_pointer(T *__TBB_atomic &)
Definition: tbb_stddef.h:309
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:739
static const intptr_t pool_bit
Definition: mailbox.h:34
affinity_id my_affinity_id
The mailbox id assigned to this scheduler.
Definition: scheduler.h:93

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::task_proxy::extract_task(), GATHER_STATISTIC, tbb::internal::no_isolation, tbb::internal::poison_pointer(), tbb::internal::task_proxy::pool_bit, and tbb::task::prefix().

Here is the call graph for this function:

◆ init_stack_info()

void tbb::internal::generic_scheduler::init_stack_info ( )

Sets up the data necessary for the stealing limiting heuristics.

Definition at line 150 of file scheduler.cpp.

150  {
151  // Stacks are growing top-down. Highest address is called "stack base",
152  // and the lowest is "stack limit".
153  __TBB_ASSERT( !my_stealing_threshold, "Stealing threshold has already been calculated" );
154  size_t stack_size = my_market->worker_stack_size();
155 #if USE_WINTHREAD
156 #if defined(_MSC_VER)&&_MSC_VER<1400 && !_WIN64
157  NT_TIB *pteb;
158  __asm mov eax, fs:[0x18]
159  __asm mov pteb, eax
160 #else
161  NT_TIB *pteb = (NT_TIB*)NtCurrentTeb();
162 #endif
163  __TBB_ASSERT( &pteb < pteb->StackBase && &pteb > pteb->StackLimit, "invalid stack info in TEB" );
164  __TBB_ASSERT( stack_size >0, "stack_size not initialized?" );
165  // When a thread is created with the attribute STACK_SIZE_PARAM_IS_A_RESERVATION, stack limit
166  // in the TIB points to the committed part of the stack only. This renders the expression
167  // "(uintptr_t)pteb->StackBase / 2 + (uintptr_t)pteb->StackLimit / 2" virtually useless.
168  // Thus for worker threads we use the explicit stack size we used while creating them.
169  // And for master threads we rely on the following fact and assumption:
170  // - the default stack size of a master thread on Windows is 1M;
171  // - if it was explicitly set by the application it is at least as large as the size of a worker stack.
172  if ( is_worker() || stack_size < MByte )
173  my_stealing_threshold = (uintptr_t)pteb->StackBase - stack_size / 2;
174  else
175  my_stealing_threshold = (uintptr_t)pteb->StackBase - MByte / 2;
176 #else /* USE_PTHREAD */
177  // There is no portable way to get stack base address in Posix, so we use
178  // non-portable method (on all modern Linux) or the simplified approach
179  // based on the common sense assumptions. The most important assumption
180  // is that the main thread's stack size is not less than that of other threads.
181  // See also comment 3 at the end of this file
182  void *stack_base = &stack_size;
183 #if __linux__ && !__bg__
184 #if __TBB_ipf
185  void *rsb_base = __TBB_get_bsp();
186 #endif
187  size_t np_stack_size = 0;
188  void *stack_limit = NULL;
189  pthread_attr_t np_attr_stack;
190  if( 0 == pthread_getattr_np(pthread_self(), &np_attr_stack) ) {
191  if ( 0 == pthread_attr_getstack(&np_attr_stack, &stack_limit, &np_stack_size) ) {
192 #if __TBB_ipf
193  pthread_attr_t attr_stack;
194  if ( 0 == pthread_attr_init(&attr_stack) ) {
195  if ( 0 == pthread_attr_getstacksize(&attr_stack, &stack_size) ) {
196  if ( np_stack_size < stack_size ) {
197  // We are in a secondary thread. Use reliable data.
198  // IA-64 architecture stack is split into RSE backup and memory parts
199  rsb_base = stack_limit;
200  stack_size = np_stack_size/2;
201  // Limit of the memory part of the stack
202  stack_limit = (char*)stack_limit + stack_size;
203  }
204  // We are either in the main thread or this thread stack
205  // is bigger that that of the main one. As we cannot discern
206  // these cases we fall back to the default (heuristic) values.
207  }
208  pthread_attr_destroy(&attr_stack);
209  }
210  // IA-64 architecture stack is split into RSE backup and memory parts
211  my_rsb_stealing_threshold = (uintptr_t)((char*)rsb_base + stack_size/2);
212 #endif /* __TBB_ipf */
213  // Size of the stack free part
214  stack_size = size_t((char*)stack_base - (char*)stack_limit);
215  }
216  pthread_attr_destroy(&np_attr_stack);
217  }
218 #endif /* __linux__ */
219  __TBB_ASSERT( stack_size>0, "stack size must be positive" );
220  my_stealing_threshold = (uintptr_t)((char*)stack_base - stack_size/2);
221 #endif /* USE_PTHREAD */
222 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
bool is_worker() const
True if running on a worker thread, false otherwise.
Definition: scheduler.h:595
market * my_market
The market I am in.
Definition: scheduler.h:159
const size_t MByte
Definition: tbb_misc.h:44
size_t worker_stack_size() const
Returns the requested stack size of worker threads.
Definition: market.h:298
void * __TBB_get_bsp()
Retrieves the current RSE backing store pointer. IA64 specific.
uintptr_t my_stealing_threshold
Position in the call stack specifying its maximal filling when stealing is still allowed.
Definition: scheduler.h:142

References __TBB_ASSERT, __TBB_get_bsp(), is_worker(), tbb::internal::MByte, my_market, my_stealing_threshold, and tbb::internal::market::worker_stack_size().

Here is the call graph for this function:

◆ is_local_task_pool_quiescent()

bool tbb::internal::generic_scheduler::is_local_task_pool_quiescent ( ) const
inline

Definition at line 555 of file scheduler.h.

555  {
557  task** tp = my_arena_slot->task_pool;
558  return tp == EmptyTaskPool || tp == LockedTaskPool;
559 }
#define LockedTaskPool
Definition: scheduler.h:47
#define EmptyTaskPool
Definition: scheduler.h:46
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task

References __TBB_ASSERT, EmptyTaskPool, LockedTaskPool, tbb::internal::scheduler_state::my_arena_slot, and tbb::internal::arena_slot_line1::task_pool.

Referenced by commit_relocated_tasks(), is_quiescent_local_task_pool_empty(), and is_quiescent_local_task_pool_reset().

Here is the caller graph for this function:

◆ is_proxy()

static bool tbb::internal::generic_scheduler::is_proxy ( const task t)
inlinestatic

True if t is a task_proxy.

Definition at line 273 of file scheduler.h.

273  {
274  return t.prefix().extra_state==es_task_proxy;
275  }
Tag for v3 task_proxy.

References tbb::internal::es_task_proxy, and tbb::task::prefix().

Referenced by steal_task(), and steal_task_from().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ is_quiescent_local_task_pool_empty()

bool tbb::internal::generic_scheduler::is_quiescent_local_task_pool_empty ( ) const
inline

Definition at line 561 of file scheduler.h.

561  {
562  __TBB_ASSERT( is_local_task_pool_quiescent(), "Task pool is not quiescent" );
564 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
__TBB_atomic size_t head
Index of the first ready task in the deque.
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
bool is_local_task_pool_quiescent() const
Definition: scheduler.h:555
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:739

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::arena_slot_line1::head, is_local_task_pool_quiescent(), tbb::internal::scheduler_state::my_arena_slot, and tbb::internal::arena_slot_line2::tail.

Referenced by leave_task_pool().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ is_quiescent_local_task_pool_reset()

bool tbb::internal::generic_scheduler::is_quiescent_local_task_pool_reset ( ) const
inline

Definition at line 566 of file scheduler.h.

566  {
567  __TBB_ASSERT( is_local_task_pool_quiescent(), "Task pool is not quiescent" );
569 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
__TBB_atomic size_t head
Index of the first ready task in the deque.
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
bool is_local_task_pool_quiescent() const
Definition: scheduler.h:555
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:739

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::arena_slot_line1::head, is_local_task_pool_quiescent(), tbb::internal::scheduler_state::my_arena_slot, and tbb::internal::arena_slot_line2::tail.

Referenced by get_task(), and prepare_task_pool().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ is_task_pool_published()

bool tbb::internal::generic_scheduler::is_task_pool_published ( ) const
inline

Definition at line 550 of file scheduler.h.

550  {
553 }
#define EmptyTaskPool
Definition: scheduler.h:46
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75

References __TBB_ASSERT, EmptyTaskPool, tbb::internal::scheduler_state::my_arena_slot, and tbb::internal::arena_slot_line1::task_pool.

Referenced by acquire_task_pool(), cleanup_master(), get_task(), leave_task_pool(), local_spawn(), prepare_task_pool(), and release_task_pool().

Here is the caller graph for this function:

◆ is_version_3_task()

static bool tbb::internal::generic_scheduler::is_version_3_task ( task t)
inlinestatic

Definition at line 133 of file scheduler.h.

133  {
134 #if __TBB_PREVIEW_CRITICAL_TASKS
135  return (t.prefix().extra_state & 0x7)>=0x1;
136 #else
137  return (t.prefix().extra_state & 0x0F)>=0x1;
138 #endif
139  }

References tbb::task::prefix().

Referenced by prepare_for_spawning(), and steal_task().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ is_worker()

bool tbb::internal::generic_scheduler::is_worker ( ) const
inline

True if running on a worker thread, false otherwise.

Definition at line 595 of file scheduler.h.

595  {
597 }
bool type
Indicates that a scheduler acts as a master or a worker.
Definition: scheduler.h:54
scheduler_properties my_properties
Definition: scheduler.h:95

References tbb::internal::scheduler_state::my_properties, tbb::internal::scheduler_properties::type, and tbb::internal::scheduler_properties::worker.

Referenced by tbb::internal::market::cleanup(), init_stack_info(), master_outermost_level(), nested_arena_entry(), nested_arena_exit(), and worker_outermost_level().

Here is the caller graph for this function:

◆ leave_task_pool()

void tbb::internal::generic_scheduler::leave_task_pool ( )
inline

Leave the task pool.

Leaving task pool automatically releases the task pool if it is locked.

Definition at line 1226 of file scheduler.cpp.

1226  {
1227  __TBB_ASSERT( is_task_pool_published(), "Not in arena" );
1228  // Do not reset my_arena_index. It will be used to (attempt to) re-acquire the slot next time
1229  __TBB_ASSERT( &my_arena->my_slots[my_arena_index] == my_arena_slot, "arena slot and slot index mismatch" );
1230  __TBB_ASSERT ( my_arena_slot->task_pool == LockedTaskPool, "Task pool must be locked when leaving arena" );
1231  __TBB_ASSERT ( is_quiescent_local_task_pool_empty(), "Cannot leave arena when the task pool is not empty" );
1233  // No release fence is necessary here as this assignment precludes external
1234  // accesses to the local task pool when becomes visible. Thus it is harmless
1235  // if it gets hoisted above preceding local bookkeeping manipulations.
1237 }
#define LockedTaskPool
Definition: scheduler.h:47
#define EmptyTaskPool
Definition: scheduler.h:46
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
arena_slot my_slots[1]
Definition: arena.h:300
size_t my_arena_index
Index of the arena slot the scheduler occupies now, or occupied last time.
Definition: scheduler.h:72
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void __TBB_store_relaxed(volatile T &location, V value)
Definition: tbb_machine.h:743
bool is_quiescent_local_task_pool_empty() const
Definition: scheduler.h:561
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, tbb::internal::__TBB_store_relaxed(), EmptyTaskPool, is_quiescent_local_task_pool_empty(), is_task_pool_published(), ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena::my_slots, sync_releasing, and tbb::internal::arena_slot_line1::task_pool.

Referenced by cleanup_master(), and reset_task_pool_and_leave().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ local_spawn()

void tbb::internal::generic_scheduler::local_spawn ( task first,
task *&  next 
)

Conceptually, this method should be a member of class scheduler. But doing so would force us to publish class scheduler in the headers.

Definition at line 618 of file scheduler.cpp.

618  {
619  __TBB_ASSERT( first, NULL );
620  __TBB_ASSERT( governor::is_set(this), NULL );
621 #if __TBB_TODO
622  // We need to consider capping the max task pool size and switching
623  // to in-place task execution whenever it is reached.
624 #endif
625  if ( &first->prefix().next == &next ) {
626  // Single task is being spawned
627 #if __TBB_TODO
628  // TODO:
629  // In the future we need to add overloaded spawn method for a single task,
630  // and a method accepting an array of task pointers (we may also want to
631  // change the implementation of the task_list class). But since such changes
632  // may affect the binary compatibility, we postpone them for a while.
633 #endif
634 #if __TBB_PREVIEW_CRITICAL_TASKS
635  if( !handled_as_critical( *first ) )
636 #endif
637  {
638  size_t T = prepare_task_pool( 1 );
640  commit_spawned_tasks( T + 1 );
641  if ( !is_task_pool_published() )
643  }
644  }
645  else {
646  // Task list is being spawned
647 #if __TBB_TODO
648  // TODO: add task_list::front() and implement&document the local execution ordering which is
649  // opposite to the current implementation. The idea is to remove hackish fast_reverse_vector
650  // and use push_back/push_front when accordingly LIFO and FIFO order of local execution is
651  // desired. It also requires refactoring of the reload_tasks method and my_offloaded_tasks list.
652  // Additional benefit may come from adding counter to the task_list so that it can reserve enough
653  // space in the task pool in advance and move all the tasks directly without any intermediate
654  // storages. But it requires dealing with backward compatibility issues and still supporting
655  // counter-less variant (though not necessarily fast implementation).
656 #endif
657  task *arr[min_task_pool_size];
658  fast_reverse_vector<task*> tasks(arr, min_task_pool_size);
659  task *t_next = NULL;
660  for( task* t = first; ; t = t_next ) {
661  // If t is affinitized to another thread, it may already be executed
662  // and destroyed by the time prepare_for_spawning returns.
663  // So milk it while it is alive.
664  bool end = &t->prefix().next == &next;
665  t_next = t->prefix().next;
666 #if __TBB_PREVIEW_CRITICAL_TASKS
667  if( !handled_as_critical( *t ) )
668 #endif
669  tasks.push_back( prepare_for_spawning(t) );
670  if( end )
671  break;
672  }
673  if( size_t num_tasks = tasks.size() ) {
674  size_t T = prepare_task_pool( num_tasks );
675  tasks.copy_memory( my_arena_slot->task_pool_ptr + T );
676  commit_spawned_tasks( T + num_tasks );
677  if ( !is_task_pool_published() )
679  }
680  }
683 }
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id __itt_string_handle ITT_FORMAT p const __itt_domain __itt_id ITT_FORMAT p const __itt_domain __itt_id __itt_timestamp __itt_timestamp end
void publish_task_pool()
Used by workers to enter the task pool.
Definition: scheduler.cpp:1214
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
auto first(Container &c) -> decltype(begin(c))
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
size_t prepare_task_pool(size_t n)
Makes sure that the task pool can accommodate at least n more elements.
Definition: scheduler.cpp:406
void commit_spawned_tasks(size_t new_tail)
Makes newly spawned tasks visible to thieves.
Definition: scheduler.h:632
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
task * prepare_for_spawning(task *t)
Checks if t is affinitized to another thread, and if so, bundles it as proxy.
Definition: scheduler.cpp:562
static bool is_set(generic_scheduler *s)
Used to check validity of the local scheduler TLS contents.
Definition: governor.cpp:124
void advertise_new_work()
If necessary, raise a flag that there is new job in arena.
Definition: arena.h:393
static const size_t min_task_pool_size
Definition: scheduler.h:294
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, tbb::internal::arena::advertise_new_work(), assert_task_pool_valid(), commit_spawned_tasks(), tbb::internal::fast_reverse_vector< T, max_segments >::copy_memory(), end, tbb::internal::first(), tbb::internal::governor::is_set(), is_task_pool_published(), min_task_pool_size, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_slot, tbb::task::prefix(), prepare_for_spawning(), prepare_task_pool(), publish_task_pool(), tbb::internal::fast_reverse_vector< T, max_segments >::push_back(), tbb::internal::fast_reverse_vector< T, max_segments >::size(), tbb::internal::arena_slot_line2::task_pool_ptr, and tbb::internal::arena::work_spawned.

Referenced by local_spawn_root_and_wait(), spawn(), and tbb::internal::custom_scheduler< SchedulerTraits >::tally_completion_of_predecessor().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ local_spawn_root_and_wait()

void tbb::internal::generic_scheduler::local_spawn_root_and_wait ( task first,
task *&  next 
)

Definition at line 685 of file scheduler.cpp.

685  {
686  __TBB_ASSERT( governor::is_set(this), NULL );
687  __TBB_ASSERT( first, NULL );
688  auto_empty_task dummy( __TBB_CONTEXT_ARG(this, first->prefix().context) );
690  for( task* t=first; ; t=t->prefix().next ) {
691  ++n;
692  __TBB_ASSERT( !t->prefix().parent, "not a root task, or already running" );
693  t->prefix().parent = &dummy;
694  if( &t->prefix().next==&next ) break;
695 #if __TBB_TASK_GROUP_CONTEXT
696  __TBB_ASSERT( t->prefix().context == t->prefix().next->prefix().context,
697  "all the root tasks in list must share the same context");
698 #endif /* __TBB_TASK_GROUP_CONTEXT */
699  }
700  dummy.prefix().ref_count = n+1;
701  if( n>1 )
702  local_spawn( first->prefix().next, next );
703  local_wait_for_all( dummy, first );
704 }
#define __TBB_CONTEXT_ARG(arg1, context)
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
auto first(Container &c) -> decltype(begin(c))
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
void local_spawn(task *first, task *&next)
Definition: scheduler.cpp:618
static bool is_set(generic_scheduler *s)
Used to check validity of the local scheduler TLS contents.
Definition: governor.cpp:124
virtual void local_wait_for_all(task &parent, task *child)=0
intptr_t reference_count
A reference count.
Definition: task.h:121

References __TBB_ASSERT, __TBB_CONTEXT_ARG, tbb::internal::first(), tbb::internal::governor::is_set(), local_spawn(), local_wait_for_all(), tbb::internal::auto_empty_task::prefix(), tbb::task::prefix(), and tbb::internal::task_prefix::ref_count.

Referenced by spawn_root_and_wait().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ local_wait_for_all()

virtual void tbb::internal::generic_scheduler::local_wait_for_all ( task parent,
task child 
)
pure virtual

Implemented in tbb::internal::custom_scheduler< SchedulerTraits >.

Referenced by cleanup_master(), local_spawn_root_and_wait(), and wait_until_empty().

Here is the caller graph for this function:

◆ lock_task_pool()

task ** tbb::internal::generic_scheduler::lock_task_pool ( arena_slot victim_arena_slot) const
inline

Locks victim's task pool, and returns pointer to it. The pointer can be NULL.

Garbles victim_arena_slot->task_pool for the duration of the lock.

ATTENTION: This method is mostly the same as generic_scheduler::acquire_task_pool(), with a little different logic of slot state checks (slot can be empty, locked or point to any task pool other than ours, and asynchronous transitions between all these states are possible). Thus if any of them is changed, consider changing the counterpart as well

Definition at line 504 of file scheduler.cpp.

504  {
505  task** victim_task_pool;
506  bool sync_prepare_done = false;
507  for( atomic_backoff backoff;; /*backoff pause embedded in the loop*/) {
508  victim_task_pool = victim_arena_slot->task_pool;
509  // NOTE: Do not use comparison of head and tail indices to check for
510  // the presence of work in the victim's task pool, as they may give
511  // incorrect indication because of task pool relocations and resizes.
512  if ( victim_task_pool == EmptyTaskPool ) {
513  // The victim thread emptied its task pool - nothing to lock
514  if( sync_prepare_done )
515  ITT_NOTIFY(sync_cancel, victim_arena_slot);
516  break;
517  }
518  if( victim_task_pool != LockedTaskPool &&
519  as_atomic(victim_arena_slot->task_pool).compare_and_swap(LockedTaskPool, victim_task_pool ) == victim_task_pool )
520  {
521  // We've locked victim's task pool
522  ITT_NOTIFY(sync_acquired, victim_arena_slot);
523  break;
524  }
525  else if( !sync_prepare_done ) {
526  // Start waiting
527  ITT_NOTIFY(sync_prepare, victim_arena_slot);
528  sync_prepare_done = true;
529  }
530  GATHER_STATISTIC( ++my_counters.thieves_conflicts );
531  // Someone else acquired a lock, so pause and do exponential backoff.
532 #if __TBB_STEALING_ABORT_ON_CONTENTION
533  if(!backoff.bounded_pause()) {
534  // the 16 was acquired empirically and a theory behind it supposes
535  // that number of threads becomes much bigger than number of
536  // tasks which can be spawned by one thread causing excessive contention.
537  // TODO: However even small arenas can benefit from the abort on contention
538  // if preemption of a thief is a problem
539  if(my_arena->my_limit >= 16)
540  return EmptyTaskPool;
541  __TBB_Yield();
542  }
543 #else
544  backoff.pause();
545 #endif
546  }
547  __TBB_ASSERT( victim_task_pool == EmptyTaskPool ||
548  (victim_arena_slot->task_pool == LockedTaskPool && victim_task_pool != LockedTaskPool),
549  "not really locked victim's task pool?" );
550  return victim_task_pool;
551 } // generic_scheduler::lock_task_pool
#define LockedTaskPool
Definition: scheduler.h:47
#define EmptyTaskPool
Definition: scheduler.h:46
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
#define __TBB_Yield()
Definition: ibm_aix51.h:48
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p sync_cancel
#define GATHER_STATISTIC(x)
atomic< unsigned > my_limit
The maximal number of currently busy slots.
Definition: arena.h:69
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
atomic< T > & as_atomic(T &t)
Definition: atomic.h:547
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, __TBB_Yield, tbb::internal::as_atomic(), EmptyTaskPool, GATHER_STATISTIC, ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena, tbb::internal::arena_base::my_limit, sync_cancel, and tbb::internal::arena_slot_line1::task_pool.

Referenced by steal_task_from().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ master_outermost_level()

bool tbb::internal::generic_scheduler::master_outermost_level ( ) const
inline

True if the scheduler is on the outermost dispatch level in a master thread.

Returns true when this scheduler instance is associated with an application thread, and is not executing any TBB task. This includes being in a TBB dispatch loop (one of wait_for_all methods) invoked directly from that thread.

Definition at line 575 of file scheduler.h.

575  {
576  return !is_worker() && outermost_level();
577 }
bool is_worker() const
True if running on a worker thread, false otherwise.
Definition: scheduler.h:595
bool outermost_level() const
True if the scheduler is on the outermost dispatch level.
Definition: scheduler.h:571

References is_worker(), and outermost_level().

Here is the call graph for this function:

◆ max_threads_in_arena()

unsigned tbb::internal::generic_scheduler::max_threads_in_arena ( )
inline

Returns the concurrency limit of the current arena.

Definition at line 599 of file scheduler.h.

599  {
600  __TBB_ASSERT(my_arena, NULL);
601  return my_arena->my_num_slots;
602 }
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
unsigned my_num_slots
The number of slots in the arena.
Definition: arena.h:153
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, tbb::internal::scheduler_state::my_arena, and tbb::internal::arena_base::my_num_slots.

Referenced by tbb::internal::get_initial_auto_partitioner_divisor(), and tbb::internal::affinity_partitioner_base_v3::resize().

Here is the caller graph for this function:

◆ nested_arena_entry()

void tbb::internal::generic_scheduler::nested_arena_entry ( arena a,
size_t  slot_index 
)

Definition at line 679 of file arena.cpp.

679  {
680  __TBB_ASSERT( is_alive(a->my_guard), NULL );
681  __TBB_ASSERT( a!=my_arena, NULL);
682 
683  // overwrite arena settings
684 #if __TBB_TASK_PRIORITY
685  if ( my_offloaded_tasks )
686  my_arena->orphan_offloaded_tasks( *this );
687  my_offloaded_tasks = NULL;
688 #endif /* __TBB_TASK_PRIORITY */
689  attach_arena( a, slot_index, /*is_master*/true );
690  __TBB_ASSERT( my_arena == a, NULL );
692  // TODO? ITT_NOTIFY(sync_acquired, a->my_slots + index);
693  // TODO: it requires market to have P workers (not P-1)
694  // TODO: a preempted worker should be excluded from assignment to other arenas e.g. my_slack--
695  if( !is_worker() && slot_index >= my_arena->my_num_reserved_slots )
697 #if __TBB_ARENA_OBSERVER
698  my_last_local_observer = 0; // TODO: try optimize number of calls
699  my_arena->my_observers.notify_entry_observers( my_last_local_observer, /*worker=*/false );
700 #endif
701 }
void attach_arena(arena *, size_t index, bool is_master)
Definition: arena.cpp:40
market * my_market
The market that owns this arena.
Definition: arena.h:135
unsigned my_num_reserved_slots
The number of reserved slots (can be occupied only by masters).
Definition: arena.h:156
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
void adjust_demand(arena &, int delta)
Request that arena's need in workers should be adjusted.
Definition: market.cpp:590
bool is_worker() const
True if running on a worker thread, false otherwise.
Definition: scheduler.h:595
static void assume_scheduler(generic_scheduler *s)
Temporarily set TLS slot to the given scheduler.
Definition: governor.cpp:120
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, tbb::internal::market::adjust_demand(), tbb::internal::governor::assume_scheduler(), attach_arena(), is_worker(), tbb::internal::scheduler_state::my_arena, tbb::internal::arena_base::my_market, and tbb::internal::arena_base::my_num_reserved_slots.

Here is the call graph for this function:

◆ nested_arena_exit()

void tbb::internal::generic_scheduler::nested_arena_exit ( )

Definition at line 703 of file arena.cpp.

703  {
704 #if __TBB_ARENA_OBSERVER
705  my_arena->my_observers.notify_exit_observers( my_last_local_observer, /*worker=*/false );
706 #endif /* __TBB_ARENA_OBSERVER */
707 #if __TBB_TASK_PRIORITY
708  if ( my_offloaded_tasks )
709  my_arena->orphan_offloaded_tasks( *this );
710 #endif
713  // Free the master slot.
714  __TBB_ASSERT(my_arena->my_slots[my_arena_index].my_scheduler, "A slot is already empty");
716  my_arena->my_exit_monitors.notify_one(); // do not relax!
717 }
market * my_market
The market that owns this arena.
Definition: arena.h:135
unsigned my_num_reserved_slots
The number of reserved slots (can be occupied only by masters).
Definition: arena.h:156
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
void adjust_demand(arena &, int delta)
Request that arena's need in workers should be adjusted.
Definition: market.cpp:590
bool is_worker() const
True if running on a worker thread, false otherwise.
Definition: scheduler.h:595
void notify_one()
Notify one thread about the event.
generic_scheduler * my_scheduler
Scheduler of the thread attached to the slot.
arena_slot my_slots[1]
Definition: arena.h:300
size_t my_arena_index
Index of the arena slot the scheduler occupies now, or occupied last time.
Definition: scheduler.h:72
concurrent_monitor my_exit_monitors
Waiting object for master threads that cannot join the arena.
Definition: arena.h:171
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:717
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), tbb::internal::market::adjust_demand(), is_worker(), tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::arena_base::my_exit_monitors, tbb::internal::arena_base::my_market, tbb::internal::arena_base::my_num_reserved_slots, tbb::internal::arena_slot_line1::my_scheduler, tbb::internal::arena::my_slots, and tbb::internal::concurrent_monitor::notify_one().

Referenced by tbb::internal::nested_arena_context::~nested_arena_context().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ outermost_level()

bool tbb::internal::generic_scheduler::outermost_level ( ) const
inline

True if the scheduler is on the outermost dispatch level.

Definition at line 571 of file scheduler.h.

571  {
572  return my_properties.outermost;
573 }
bool outermost
Indicates that a scheduler is on outermost level.
Definition: scheduler.h:57
scheduler_properties my_properties
Definition: scheduler.h:95

References tbb::internal::scheduler_state::my_properties, and tbb::internal::scheduler_properties::outermost.

Referenced by master_outermost_level(), and worker_outermost_level().

Here is the caller graph for this function:

◆ plugged_return_list()

static task* tbb::internal::generic_scheduler::plugged_return_list ( )
inlinestatic

Special value used to mark my_return_list as not taking any more entries.

Definition at line 380 of file scheduler.h.

380 {return (task*)(intptr_t)(-1);}
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task

Referenced by free_nonlocal_small_task(), and free_scheduler().

Here is the caller graph for this function:

◆ prepare_for_spawning()

task * tbb::internal::generic_scheduler::prepare_for_spawning ( task t)
inline

Checks if t is affinitized to another thread, and if so, bundles it as proxy.

Returns either t or proxy containing t.

Definition at line 562 of file scheduler.cpp.

562  {
563  __TBB_ASSERT( t->state()==task::allocated, "attempt to spawn task that is not in 'allocated' state" );
564  t->prefix().state = task::ready;
565 #if TBB_USE_ASSERT
566  if( task* parent = t->parent() ) {
567  internal::reference_count ref_count = parent->prefix().ref_count;
568  __TBB_ASSERT( ref_count>=0, "attempt to spawn task whose parent has a ref_count<0" );
569  __TBB_ASSERT( ref_count!=0, "attempt to spawn task whose parent has a ref_count==0 (forgot to set_ref_count?)" );
570  parent->prefix().extra_state |= es_ref_count_active;
571  }
572 #endif /* TBB_USE_ASSERT */
573  affinity_id dst_thread = t->prefix().affinity;
574  __TBB_ASSERT( dst_thread == 0 || is_version_3_task(*t),
575  "backwards compatibility to TBB 2.0 tasks is broken" );
576 #if __TBB_TASK_ISOLATION
577  isolation_tag isolation = my_innermost_running_task->prefix().isolation;
578  t->prefix().isolation = isolation;
579 #endif /* __TBB_TASK_ISOLATION */
580  if( dst_thread != 0 && dst_thread != my_affinity_id ) {
581  task_proxy& proxy = (task_proxy&)allocate_task( sizeof(task_proxy),
582  __TBB_CONTEXT_ARG(NULL, NULL) );
583  // Mark as a proxy
584  proxy.prefix().extra_state = es_task_proxy;
585  proxy.outbox = &my_arena->mailbox(dst_thread);
586  // Mark proxy as present in both locations (sender's task pool and destination mailbox)
587  proxy.task_and_tag = intptr_t(t) | task_proxy::location_mask;
588 #if __TBB_TASK_PRIORITY
589  poison_pointer( proxy.prefix().context );
590 #endif /* __TBB_TASK_PRIORITY */
591  __TBB_ISOLATION_EXPR( proxy.prefix().isolation = isolation );
592  ITT_NOTIFY( sync_releasing, proxy.outbox );
593  // Mail the proxy - after this point t may be destroyed by another thread at any moment.
594  proxy.outbox->push(&proxy);
595  return &proxy;
596  }
597  return t;
598 }
#define __TBB_CONTEXT_ARG(arg1, context)
unsigned short affinity_id
An id as used for specifying affinity.
Definition: task.h:124
Set if ref_count might be changed by another thread. Used for debugging.
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
task object is freshly allocated or recycled.
Definition: task.h:620
#define __TBB_ISOLATION_EXPR(isolation)
task * my_innermost_running_task
Innermost task whose task::execute() is running. A dummy task on the outermost level.
Definition: scheduler.h:81
Tag for v3 task_proxy.
intptr_t isolation_tag
A tag for task isolation.
Definition: task.h:128
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
static const intptr_t location_mask
Definition: mailbox.h:36
static bool is_version_3_task(task &t)
Definition: scheduler.h:133
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t int ITT_FORMAT d no args no args unsigned int ITT_FORMAT u const __itt_domain __itt_id ITT_FORMAT lu const __itt_domain __itt_id __itt_id parent
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
task is in ready pool, or is going to be put there, or was just taken off.
Definition: task.h:618
void poison_pointer(T *__TBB_atomic &)
Definition: tbb_stddef.h:309
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
mail_outbox & mailbox(affinity_id id)
Get reference to mailbox corresponding to given affinity_id.
Definition: arena.h:208
task & allocate_task(size_t number_of_bytes, __TBB_CONTEXT_ARG(task *parent, task_group_context *context))
Allocate task object, either from the heap or a free list.
Definition: scheduler.cpp:304
intptr_t reference_count
A reference count.
Definition: task.h:121
internal::task_prefix & prefix(internal::version_tag *=NULL) const
Get reference to corresponding task_prefix.
Definition: task.h:941
affinity_id my_affinity_id
The mailbox id assigned to this scheduler.
Definition: scheduler.h:93
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, __TBB_CONTEXT_ARG, __TBB_ISOLATION_EXPR, allocate_task(), tbb::task::allocated, tbb::task::context(), tbb::internal::es_ref_count_active, tbb::internal::es_task_proxy, is_version_3_task(), ITT_NOTIFY, tbb::internal::task_proxy::location_mask, tbb::internal::arena::mailbox(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_innermost_running_task, tbb::internal::task_proxy::outbox, parent, tbb::task::parent(), tbb::internal::poison_pointer(), tbb::task::prefix(), tbb::internal::mail_outbox::push(), tbb::task::ready, tbb::task::state(), sync_releasing, and tbb::internal::task_proxy::task_and_tag.

Referenced by local_spawn().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ prepare_task_pool()

size_t tbb::internal::generic_scheduler::prepare_task_pool ( size_t  n)
inline

Makes sure that the task pool can accommodate at least n more elements.

If necessary relocates existing task pointers or grows the ready task deque. Returns (possible updated) tail index (not accounting for n).

Definition at line 406 of file scheduler.cpp.

406  {
407  size_t T = __TBB_load_relaxed(my_arena_slot->tail); // mirror
408  if ( T + num_tasks <= my_arena_slot->my_task_pool_size )
409  return T;
410 
411  size_t new_size = num_tasks;
412 
416  if ( num_tasks < min_task_pool_size ) new_size = min_task_pool_size;
418  return 0;
419  }
420 
422  size_t H = __TBB_load_relaxed( my_arena_slot->head ); // mirror
423  task** task_pool = my_arena_slot->task_pool_ptr;;
425  // Count not skipped tasks. Consider using std::count_if.
426  for ( size_t i = H; i < T; ++i )
427  if ( task_pool[i] ) ++new_size;
428  // If the free space at the beginning of the task pool is too short, we
429  // are likely facing a pathological single-producer-multiple-consumers
430  // scenario, and thus it's better to expand the task pool
432  if ( allocate ) {
433  // Grow task pool. As this operation is rare, and its cost is asymptotically
434  // amortizable, we can tolerate new task pool allocation done under the lock.
435  if ( new_size < 2 * my_arena_slot->my_task_pool_size )
437  my_arena_slot->allocate_task_pool( new_size ); // updates my_task_pool_size
438  }
439  // Filter out skipped tasks. Consider using std::copy_if.
440  size_t T1 = 0;
441  for ( size_t i = H; i < T; ++i )
442  if ( task_pool[i] )
443  my_arena_slot->task_pool_ptr[T1++] = task_pool[i];
444  // Deallocate the previous task pool if a new one has been allocated.
445  if ( allocate )
446  NFS_Free( task_pool );
447  else
449  // Publish the new state.
452  return T1;
453 }
void fill_with_canary_pattern(size_t, size_t)
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
bool is_quiescent_local_task_pool_reset() const
Definition: scheduler.h:566
void allocate_task_pool(size_t n)
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task __itt_model_task_instance ITT_FORMAT p void ITT_FORMAT p void ITT_FORMAT p void size_t ITT_FORMAT d void ITT_FORMAT p const wchar_t ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s const char ITT_FORMAT s no args void ITT_FORMAT p size_t ITT_FORMAT d no args const wchar_t const wchar_t ITT_FORMAT s __itt_heap_function void size_t int ITT_FORMAT d __itt_heap_function void ITT_FORMAT p __itt_heap_function void void size_t new_size
__TBB_atomic size_t head
Index of the first ready task in the deque.
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
void commit_relocated_tasks(size_t new_tail)
Makes relocated tasks visible to thieves and releases the local task pool.
Definition: scheduler.h:641
void acquire_task_pool() const
Locks the local task pool.
Definition: scheduler.cpp:460
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:739
void __TBB_EXPORTED_FUNC NFS_Free(void *)
Free memory allocated by NFS_Allocate.
size_t my_task_pool_size
Capacity of the primary task pool (number of elements - pointers to task).
static const size_t min_task_pool_size
Definition: scheduler.h:294

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), acquire_task_pool(), tbb::internal::arena_slot::allocate_task_pool(), assert_task_pool_valid(), commit_relocated_tasks(), tbb::internal::arena_slot::fill_with_canary_pattern(), tbb::internal::arena_slot_line1::head, is_quiescent_local_task_pool_reset(), is_task_pool_published(), min_task_pool_size, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena_slot_line2::my_task_pool_size, new_size, tbb::internal::NFS_Free(), tbb::internal::arena_slot_line2::tail, and tbb::internal::arena_slot_line2::task_pool_ptr.

Referenced by local_spawn().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ publish_task_pool()

void tbb::internal::generic_scheduler::publish_task_pool ( )
inline

Used by workers to enter the task pool.

Does not lock the task pool in case if arena slot has been successfully grabbed.

Definition at line 1214 of file scheduler.cpp.

1214  {
1215  __TBB_ASSERT ( my_arena, "no arena: initialization not completed?" );
1216  __TBB_ASSERT ( my_arena_index < my_arena->my_num_slots, "arena slot index is out-of-bound" );
1218  __TBB_ASSERT ( my_arena_slot->task_pool == EmptyTaskPool, "someone else grabbed my arena slot?" );
1220  "entering arena without tasks to share" );
1221  // Release signal on behalf of previously spawned tasks (when this thread was not in arena yet)
1224 }
#define EmptyTaskPool
Definition: scheduler.h:46
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
__TBB_atomic size_t head
Index of the first ready task in the deque.
arena_slot my_slots[1]
Definition: arena.h:300
size_t my_arena_index
Index of the arena slot the scheduler occupies now, or occupied last time.
Definition: scheduler.h:72
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:739
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:717
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_with_release(), EmptyTaskPool, tbb::internal::arena_slot_line1::head, ITT_NOTIFY, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena::my_slots, sync_releasing, tbb::internal::arena_slot_line2::tail, tbb::internal::arena_slot_line1::task_pool, and tbb::internal::arena_slot_line2::task_pool_ptr.

Referenced by get_task(), and local_spawn().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ receive_or_steal_task()

virtual task* tbb::internal::generic_scheduler::receive_or_steal_task ( __TBB_ISOLATION_ARG(__TBB_atomic reference_count &completion_ref_count, isolation_tag isolation)  )
pure virtual

Try getting a task from other threads (via mailbox, stealing, FIFO queue, orphans adoption).

Returns obtained task or NULL if all attempts fail.

Implemented in tbb::internal::custom_scheduler< SchedulerTraits >.

◆ release_task_pool()

void tbb::internal::generic_scheduler::release_task_pool ( ) const
inline

Unlocks the local task pool.

Restores my_arena_slot->task_pool munged by acquire_task_pool. Requires correctly set my_arena_slot->task_pool_ptr.

Definition at line 489 of file scheduler.cpp.

489  {
490  if ( !is_task_pool_published() )
491  return; // we are not in arena - nothing to unlock
492  __TBB_ASSERT( my_arena_slot, "we are not in arena" );
493  __TBB_ASSERT( my_arena_slot->task_pool == LockedTaskPool, "arena slot is not locked" );
496 }
#define LockedTaskPool
Definition: scheduler.h:47
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:717

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), is_task_pool_published(), ITT_NOTIFY, LockedTaskPool, tbb::internal::scheduler_state::my_arena_slot, sync_releasing, tbb::internal::arena_slot_line1::task_pool, and tbb::internal::arena_slot_line2::task_pool_ptr.

Referenced by cleanup_master(), commit_relocated_tasks(), and get_task().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ reset_task_pool_and_leave()

void tbb::internal::generic_scheduler::reset_task_pool_and_leave ( )
inline

Resets head and tail indices to 0, and leaves task pool.

The task pool must be locked by the owner (via acquire_task_pool).

Definition at line 624 of file scheduler.h.

624  {
625  __TBB_ASSERT( my_arena_slot->task_pool == LockedTaskPool, "Task pool must be locked when resetting task pool" );
628  leave_task_pool();
629 }
#define LockedTaskPool
Definition: scheduler.h:47
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
__TBB_atomic size_t head
Index of the first ready task in the deque.
arena_slot * my_arena_slot
Pointer to the slot in the arena we own at the moment.
Definition: scheduler.h:75
void __TBB_store_relaxed(volatile T &location, V value)
Definition: tbb_machine.h:743
__TBB_atomic size_t tail
Index of the element following the last ready task in the deque.
void leave_task_pool()
Leave the task pool.
Definition: scheduler.cpp:1226

References __TBB_ASSERT, tbb::internal::__TBB_store_relaxed(), tbb::internal::arena_slot_line1::head, leave_task_pool(), LockedTaskPool, tbb::internal::scheduler_state::my_arena_slot, tbb::internal::arena_slot_line2::tail, and tbb::internal::arena_slot_line1::task_pool.

Referenced by get_task().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ spawn()

void tbb::internal::generic_scheduler::spawn ( task first,
task *&  next 
)
virtual

For internal use only.

Implements tbb::internal::scheduler.

Definition at line 706 of file scheduler.cpp.

706  {
708 }
static generic_scheduler * local_scheduler()
Obtain the thread-local instance of the TBB scheduler.
Definition: governor.h:126
auto first(Container &c) -> decltype(begin(c))
void local_spawn(task *first, task *&next)
Definition: scheduler.cpp:618

References tbb::internal::first(), tbb::internal::governor::local_scheduler(), and local_spawn().

Here is the call graph for this function:

◆ spawn_root_and_wait()

void tbb::internal::generic_scheduler::spawn_root_and_wait ( task first,
task *&  next 
)
virtual

For internal use only.

Implements tbb::internal::scheduler.

Definition at line 710 of file scheduler.cpp.

710  {
712 }
static generic_scheduler * local_scheduler()
Obtain the thread-local instance of the TBB scheduler.
Definition: governor.h:126
auto first(Container &c) -> decltype(begin(c))
void local_spawn_root_and_wait(task *first, task *&next)
Definition: scheduler.cpp:685

References tbb::internal::first(), tbb::internal::governor::local_scheduler(), and local_spawn_root_and_wait().

Here is the call graph for this function:

◆ steal_task()

task * tbb::internal::generic_scheduler::steal_task ( __TBB_ISOLATION_EXPR(isolation_tag isolation)  )

Attempts to steal a task from a randomly chosen thread/scheduler.

Definition at line 1075 of file scheduler.cpp.

1075  {
1076  // Try to steal a task from a random victim.
1077  size_t k = my_random.get() % (my_arena->my_limit-1);
1078  arena_slot* victim = &my_arena->my_slots[k];
1079  // The following condition excludes the master that might have
1080  // already taken our previous place in the arena from the list .
1081  // of potential victims. But since such a situation can take
1082  // place only in case of significant oversubscription, keeping
1083  // the checks simple seems to be preferable to complicating the code.
1084  if( k >= my_arena_index )
1085  ++victim; // Adjusts random distribution to exclude self
1086  task **pool = victim->task_pool;
1087  task *t = NULL;
1088  if( pool == EmptyTaskPool || !(t = steal_task_from( __TBB_ISOLATION_ARG(*victim, isolation) )) )
1089  return NULL;
1090  if( is_proxy(*t) ) {
1091  task_proxy &tp = *(task_proxy*)t;
1092  t = tp.extract_task<task_proxy::pool_bit>();
1093  if ( !t ) {
1094  // Proxy was empty, so it's our responsibility to free it
1095  free_task<no_cache_small_task>(tp);
1096  return NULL;
1097  }
1098  GATHER_STATISTIC( ++my_counters.proxies_stolen );
1099  }
1100  t->prefix().extra_state |= es_task_is_stolen;
1101  if( is_version_3_task(*t) ) {
1103  t->prefix().owner = this;
1104  t->note_affinity( my_affinity_id );
1105  }
1106  GATHER_STATISTIC( ++my_counters.steals_committed );
1107  return t;
1108 }
#define EmptyTaskPool
Definition: scheduler.h:46
task * steal_task_from(__TBB_ISOLATION_ARG(arena_slot &victim_arena_slot, isolation_tag isolation))
Steal task from another scheduler's ready pool.
Definition: scheduler.cpp:1110
FastRandom my_random
Random number generator used for picking a random victim from which to steal.
Definition: scheduler.h:162
task * my_innermost_running_task
Innermost task whose task::execute() is running. A dummy task on the outermost level.
Definition: scheduler.h:81
Set if the task has been stolen.
arena_slot my_slots[1]
Definition: arena.h:300
size_t my_arena_index
Index of the arena slot the scheduler occupies now, or occupied last time.
Definition: scheduler.h:72
#define __TBB_ISOLATION_ARG(arg1, isolation)
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
static bool is_version_3_task(task &t)
Definition: scheduler.h:133
unsigned short get()
Get a random number.
Definition: tbb_misc.h:143
#define GATHER_STATISTIC(x)
static bool is_proxy(const task &t)
True if t is a task_proxy.
Definition: scheduler.h:273
atomic< unsigned > my_limit
The maximal number of currently busy slots.
Definition: arena.h:69
static const intptr_t pool_bit
Definition: mailbox.h:34
internal::task_prefix & prefix(internal::version_tag *=NULL) const
Get reference to corresponding task_prefix.
Definition: task.h:941
affinity_id my_affinity_id
The mailbox id assigned to this scheduler.
Definition: scheduler.h:93
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ISOLATION_ARG, EmptyTaskPool, tbb::internal::es_task_is_stolen, tbb::internal::task_proxy::extract_task(), GATHER_STATISTIC, tbb::internal::FastRandom::get(), is_proxy(), is_version_3_task(), tbb::internal::scheduler_state::my_affinity_id, tbb::internal::scheduler_state::my_arena, tbb::internal::scheduler_state::my_arena_index, tbb::internal::scheduler_state::my_innermost_running_task, tbb::internal::arena_base::my_limit, my_random, tbb::internal::arena::my_slots, tbb::task::note_affinity(), tbb::internal::task_proxy::pool_bit, tbb::task::prefix(), steal_task_from(), and tbb::internal::arena_slot_line1::task_pool.

Here is the call graph for this function:

◆ steal_task_from()

task * tbb::internal::generic_scheduler::steal_task_from ( __TBB_ISOLATION_ARG(arena_slot &victim_arena_slot, isolation_tag isolation)  )

Steal task from another scheduler's ready pool.

Definition at line 1110 of file scheduler.cpp.

1110  {
1111  task** victim_pool = lock_task_pool( &victim_slot );
1112  if ( !victim_pool )
1113  return NULL;
1114  task* result = NULL;
1115  size_t H = __TBB_load_relaxed(victim_slot.head); // mirror
1116  size_t H0 = H;
1117  bool tasks_omitted = false;
1118  do {
1119  __TBB_store_relaxed( victim_slot.head, ++H );
1120  atomic_fence();
1121  if ( (intptr_t)H > (intptr_t)__TBB_load_relaxed( victim_slot.tail ) ) {
1122  // Stealing attempt failed, deque contents has not been changed by us
1123  GATHER_STATISTIC( ++my_counters.thief_backoffs );
1124  __TBB_store_relaxed( victim_slot.head, /*dead: H = */ H0 );
1125  __TBB_ASSERT( !result, NULL );
1126  goto unlock;
1127  }
1128  __TBB_control_consistency_helper(); // on victim_slot.tail
1129  result = victim_pool[H-1];
1130  __TBB_ASSERT( !is_poisoned( result ), NULL );
1131 
1132  if ( result ) {
1133  __TBB_ISOLATION_EXPR( if ( isolation == no_isolation || isolation == result->prefix().isolation ) )
1134  {
1135  if ( !is_proxy( *result ) )
1136  break;
1137  task_proxy& tp = *static_cast<task_proxy*>(result);
1138  // If mailed task is likely to be grabbed by its destination thread, skip it.
1139  if ( !(task_proxy::is_shared( tp.task_and_tag ) && tp.outbox->recipient_is_idle()) )
1140  break;
1141  GATHER_STATISTIC( ++my_counters.proxies_bypassed );
1142  }
1143  // The task cannot be executed either due to isolation or proxy contraints.
1144  result = NULL;
1145  tasks_omitted = true;
1146  } else if ( !tasks_omitted ) {
1147  // Cleanup the task pool from holes until a task is skipped.
1148  __TBB_ASSERT( H0 == H-1, NULL );
1149  poison_pointer( victim_pool[H0] );
1150  H0 = H;
1151  }
1152  } while ( !result );
1153  __TBB_ASSERT( result, NULL );
1154 
1155  // emit "task was consumed" signal
1156  ITT_NOTIFY( sync_acquired, (void*)((uintptr_t)&victim_slot+sizeof( uintptr_t )) );
1157  poison_pointer( victim_pool[H-1] );
1158  if ( tasks_omitted ) {
1159  // Some proxies in the task pool have been omitted. Set the stolen task to NULL.
1160  victim_pool[H-1] = NULL;
1161  __TBB_store_relaxed( victim_slot.head, /*dead: H = */ H0 );
1162  }
1163 unlock:
1164  unlock_task_pool( &victim_slot, victim_pool );
1165 #if __TBB_PREFETCHING
1166  __TBB_cl_evict(&victim_slot.head);
1167  __TBB_cl_evict(&victim_slot.tail);
1168 #endif
1169  if ( tasks_omitted )
1170  // Synchronize with snapshot as the head and tail can be bumped which can falsely trigger EMPTY state
1172  return result;
1173 }
static bool is_shared(intptr_t tat)
True if the proxy is stored both in its sender's pool and in the destination mailbox.
Definition: mailbox.h:50
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
const isolation_tag no_isolation
Definition: task.h:129
#define __TBB_ISOLATION_EXPR(isolation)
task ** lock_task_pool(arena_slot *victim_arena_slot) const
Locks victim's task pool, and returns pointer to it. The pointer can be NULL.
Definition: scheduler.cpp:504
void unlock_task_pool(arena_slot *victim_arena_slot, task **victim_task_pool) const
Unlocks victim's task pool.
Definition: scheduler.cpp:553
void atomic_fence()
Sequentially consistent full memory fence.
Definition: tbb_machine.h:343
void __TBB_store_relaxed(volatile T &location, V value)
Definition: tbb_machine.h:743
#define __TBB_cl_evict(p)
Definition: mic_common.h:38
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p void ITT_FORMAT p no args __itt_suppress_mode_t unsigned int void size_t ITT_FORMAT d void ITT_FORMAT p void ITT_FORMAT p __itt_model_site __itt_model_site_instance ITT_FORMAT p __itt_model_task * task
#define GATHER_STATISTIC(x)
static bool is_proxy(const task &t)
True if t is a task_proxy.
Definition: scheduler.h:273
void poison_pointer(T *__TBB_atomic &)
Definition: tbb_stddef.h:309
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
T __TBB_load_relaxed(const volatile T &location)
Definition: tbb_machine.h:739
#define __TBB_control_consistency_helper()
Definition: gcc_generic.h:64
void advertise_new_work()
If necessary, raise a flag that there is new job in arena.
Definition: arena.h:393
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References __TBB_ASSERT, __TBB_cl_evict, __TBB_control_consistency_helper, __TBB_ISOLATION_EXPR, tbb::internal::__TBB_load_relaxed(), tbb::internal::__TBB_store_relaxed(), tbb::internal::arena::advertise_new_work(), tbb::atomic_fence(), GATHER_STATISTIC, tbb::internal::arena_slot_line1::head, is_proxy(), tbb::internal::task_proxy::is_shared(), ITT_NOTIFY, lock_task_pool(), tbb::internal::scheduler_state::my_arena, tbb::internal::no_isolation, tbb::internal::task_proxy::outbox, tbb::internal::poison_pointer(), tbb::task::prefix(), tbb::internal::mail_outbox::recipient_is_idle(), tbb::internal::arena_slot_line2::tail, tbb::internal::task_proxy::task_and_tag, unlock_task_pool(), and tbb::internal::arena::wakeup.

Referenced by steal_task().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ unlock_task_pool()

void tbb::internal::generic_scheduler::unlock_task_pool ( arena_slot victim_arena_slot,
task **  victim_task_pool 
) const
inline

Unlocks victim's task pool.

Restores victim_arena_slot->task_pool munged by lock_task_pool.

Definition at line 553 of file scheduler.cpp.

554  {
555  __TBB_ASSERT( victim_arena_slot, "empty victim arena slot pointer" );
556  __TBB_ASSERT( victim_arena_slot->task_pool == LockedTaskPool, "victim arena slot is not locked" );
557  ITT_NOTIFY(sync_releasing, victim_arena_slot);
558  __TBB_store_with_release( victim_arena_slot->task_pool, victim_task_pool );
559 }
#define LockedTaskPool
Definition: scheduler.h:47
#define __TBB_ASSERT(predicate, comment)
No-op version of __TBB_ASSERT.
Definition: tbb_stddef.h:169
void const char const char int ITT_FORMAT __itt_group_sync x void const char ITT_FORMAT __itt_group_sync s void ITT_FORMAT __itt_group_sync p void ITT_FORMAT p sync_releasing
#define ITT_NOTIFY(name, obj)
Definition: itt_notify.h:120
void __TBB_store_with_release(volatile T &location, V value)
Definition: tbb_machine.h:717

References __TBB_ASSERT, tbb::internal::__TBB_store_with_release(), ITT_NOTIFY, LockedTaskPool, sync_releasing, and tbb::internal::arena_slot_line1::task_pool.

Referenced by steal_task_from().

Here is the call graph for this function:
Here is the caller graph for this function:

◆ wait_until_empty()

void tbb::internal::generic_scheduler::wait_until_empty ( )

Definition at line 719 of file arena.cpp.

719  {
720  my_dummy_task->prefix().ref_count++; // prevents exit from local_wait_for_all when local work is done enforcing the stealing
723  my_dummy_task->prefix().ref_count--;
724 }
tbb::atomic< uintptr_t > my_pool_state
Current task pool state and estimate of available tasks amount.
Definition: arena.h:103
static const pool_state_t SNAPSHOT_EMPTY
No tasks to steal since last snapshot was taken.
Definition: arena.h:221
task * my_dummy_task
Fake root task created by slave threads.
Definition: scheduler.h:173
virtual void local_wait_for_all(task &parent, task *child)=0
internal::task_prefix & prefix(internal::version_tag *=NULL) const
Get reference to corresponding task_prefix.
Definition: task.h:941
arena * my_arena
The arena that I own (if master) or am servicing at the moment (if worker)
Definition: scheduler.h:78

References local_wait_for_all(), tbb::internal::scheduler_state::my_arena, my_dummy_task, tbb::internal::arena_base::my_pool_state, tbb::task::prefix(), and tbb::internal::arena::SNAPSHOT_EMPTY.

Here is the call graph for this function:

◆ worker_outermost_level()

bool tbb::internal::generic_scheduler::worker_outermost_level ( ) const
inline

True if the scheduler is on the outermost dispatch level in a worker thread.

Definition at line 579 of file scheduler.h.

579  {
580  return is_worker() && outermost_level();
581 }
bool is_worker() const
True if running on a worker thread, false otherwise.
Definition: scheduler.h:595
bool outermost_level() const
True if the scheduler is on the outermost dispatch level.
Definition: scheduler.h:571

References is_worker(), and outermost_level().

Here is the call graph for this function:

Friends And Related Function Documentation

◆ custom_scheduler

template<typename SchedulerTraits >
friend class custom_scheduler
friend

Definition at line 314 of file scheduler.h.

Member Data Documentation

◆ min_task_pool_size

const size_t tbb::internal::generic_scheduler::min_task_pool_size = 64
static

Initial size of the task deque sufficient to serve without reallocation 4 nested parallel_for calls with iteration space of 65535 grains each.

Definition at line 294 of file scheduler.h.

Referenced by local_spawn(), and prepare_task_pool().

◆ my_auto_initialized

bool tbb::internal::generic_scheduler::my_auto_initialized

True if *this was created by automatic TBB initialization.

Definition at line 184 of file scheduler.h.

◆ my_dummy_task

task* tbb::internal::generic_scheduler::my_dummy_task

Fake root task created by slave threads.

The task is used as the "parent" argument to method wait_for_all.

Definition at line 173 of file scheduler.h.

Referenced by attach_arena(), cleanup_master(), free_scheduler(), generic_scheduler(), tbb::internal::nested_arena_context::mimic_outermost_level(), wait_until_empty(), and tbb::internal::nested_arena_context::~nested_arena_context().

◆ my_free_list

task* tbb::internal::generic_scheduler::my_free_list

Free list of small tasks that can be reused.

Definition at line 165 of file scheduler.h.

Referenced by allocate_task(), free_scheduler(), and free_task().

◆ my_market

market* tbb::internal::generic_scheduler::my_market

The market I am in.

Definition at line 159 of file scheduler.h.

Referenced by attach_arena(), cleanup_master(), free_scheduler(), and init_stack_info().

◆ my_random

FastRandom tbb::internal::generic_scheduler::my_random

Random number generator used for picking a random victim from which to steal.

Definition at line 162 of file scheduler.h.

Referenced by steal_task(), and tbb::internal::custom_scheduler< SchedulerTraits >::tally_completion_of_predecessor().

◆ my_ref_count

long tbb::internal::generic_scheduler::my_ref_count

Reference count for scheduler.

Number of task_scheduler_init objects that point to this scheduler

Definition at line 177 of file scheduler.h.

◆ my_return_list

task* tbb::internal::generic_scheduler::my_return_list

List of small tasks that have been returned to this scheduler by other schedulers.

Definition at line 387 of file scheduler.h.

Referenced by allocate_task(), free_scheduler(), and generic_scheduler().

◆ my_small_task_count

__TBB_atomic intptr_t tbb::internal::generic_scheduler::my_small_task_count

Number of small tasks that have been allocated by this scheduler.

Definition at line 383 of file scheduler.h.

Referenced by allocate_task(), and free_scheduler().

◆ my_stealing_threshold

uintptr_t tbb::internal::generic_scheduler::my_stealing_threshold

Position in the call stack specifying its maximal filling when stealing is still allowed.

Definition at line 142 of file scheduler.h.

Referenced by can_steal(), and init_stack_info().

◆ null_arena_index

const size_t tbb::internal::generic_scheduler::null_arena_index = ~size_t(0)
static

Definition at line 148 of file scheduler.h.

◆ quick_task_size

const size_t tbb::internal::generic_scheduler::quick_task_size = 256-task_prefix_reservation_size
static

If sizeof(task) is <=quick_task_size, it is handled on a free list instead of malloc'd.

Definition at line 131 of file scheduler.h.

Referenced by allocate_task().


The documentation for this class was generated from the following files:

Copyright © 2005-2019 Intel Corporation. All Rights Reserved.

Intel, Pentium, Intel Xeon, Itanium, Intel XScale and VTune are registered trademarks or trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

* Other names and brands may be claimed as the property of others.