OPeNDAP Hyrax Back End Server (BES)  Updated for version 3.8.3
BESStoredDapResultCache Class Reference

This class is used to cache DAP2 response objects. More...

#include <BESStoredDapResultCache.h>

Inheritance diagram for BESStoredDapResultCache:
Inheritance graph
Collaboration diagram for BESStoredDapResultCache:
Collaboration graph

Public Member Functions

virtual libdap::DDS * cache_dataset (libdap::DDS &dds, const std::string &constraint, BESDapResponseBuilder *rb, libdap::ConstraintEvaluator *eval, std::string &cache_token)
 Get the cached DDS object. More...
 
virtual bool cache_too_big (unsigned long long current_size) const
 look at the cache size; is it too large? Look at the cache size and see if it is too big. More...
 
virtual bool create_and_lock (const string &target, int &fd)
 Create a file in the cache and lock it for write access. More...
 
virtual void dump (ostream &strm) const
 dumps information about this object More...
 
virtual void exclusive_to_shared_lock (int fd)
 Transfer from an exclusive lock to a shared lock. More...
 
virtual string get_cache_file_name (const string &src, bool mangle=false)
 Build the name of file that will holds the uncompressed data from 'src' in the cache. More...
 
virtual unsigned long long get_cache_size ()
 Get the cache size. More...
 
virtual bool get_read_lock (const string &target, int &fd)
 Get a read-only lock on the file if it exists. More...
 
const string getCacheDirectory ()
 
const string getCacheFilePrefix ()
 
virtual void lock_cache_read ()
 Get a shared lock on the 'cache info' file. More...
 
virtual void lock_cache_write ()
 Get an exclusive lock on the 'cache info' file. More...
 
virtual void purge_file (const string &file)
 Purge a single file from the cache. More...
 
virtual void unlock_and_close (const string &target)
 Unlock the named file. More...
 
virtual void unlock_and_close (int fd)
 Unlock the file. More...
 
virtual void unlock_cache ()
 Unlock the cache info file. More...
 
virtual void update_and_purge (const string &new_file)
 Purge files from the cache. More...
 
virtual unsigned long long update_cache_info (const string &target)
 Update the cache info file to include 'target'. More...
 
virtual ~BESStoredDapResultCache ()
 

Static Public Member Functions

static BESStoredDapResultCacheget_instance (const string &cache_dir, const string &prefix, unsigned long long size)
 Get an instance of the BESStoreResultCache object. More...
 
static BESStoredDapResultCacheget_instance ()
 Get the default instance of the BESStoreResultCache object. More...
 
static unsigned long getCacheSizeFromConfig ()
 
static string getResultPrefixFromConfig ()
 
static string getStoredResultsDirFromConfig ()
 
static string getSubDirFromConfig ()
 

Static Public Attributes

static const string PREFIX_KEY = "DAP.StoredResultsCache.prefix"
 
static const string SIZE_KEY = "DAP.StoredResultsCache.size"
 
static const string SUBDIR_KEY = "DAP.StoredResultsCache.subdir"
 

Protected Member Functions

 BESStoredDapResultCache (const string &cache_dir, const string &prefix, unsigned long long size)
 Protected constructor that takes as arguments keys to the cache directory, file prefix, and size of the cache to be looked up a configuration file. More...
 
void initialize (const string &cache_dir, const string &prefix, unsigned long long size)
 

Friends

class StoredResultTest
 

Detailed Description

This class is used to cache DAP2 response objects.

Author
jhrg 5/3/13

Definition at line 49 of file BESStoredDapResultCache.h.

Constructor & Destructor Documentation

BESStoredDapResultCache::BESStoredDapResultCache ( const string &  stored_results_dir,
const string &  prefix,
unsigned long long  size 
)
protected

Protected constructor that takes as arguments keys to the cache directory, file prefix, and size of the cache to be looked up a configuration file.

The keys specified are looked up in the specified keys object. If not found or not set correctly then an exception is thrown. I.E., if the cache directory is empty, the size is zero, or the prefix is empty.

Parameters
stored_results_dirkey to look up in the keys file to find cache dir
prefix_keykey to look up in the keys file to find the cache prefix
size_keykey to look up in the keys file to find the cache size (in MBytes)
Exceptions
BESSyntaxUserErrorif keys not set, cache dir or prefix empty, size is 0, or if cache dir does not exist.

Definition at line 198 of file BESStoredDapResultCache.cc.

virtual BESStoredDapResultCache::~BESStoredDapResultCache ( )
inlinevirtual

Definition at line 83 of file BESStoredDapResultCache.h.

Member Function Documentation

DDS * BESStoredDapResultCache::cache_dataset ( libdap::DDS &  dds,
const std::string &  constraint,
BESDapResponseBuilder rb,
libdap::ConstraintEvaluator *  eval,
std::string &  cache_token 
)
virtual

Get the cached DDS object.

Unlike the DAP2 DDS response, this is a C++ DDS object that holds the response's variables, their attributes and their data. The DDS built by the handlers (but without data) is passed into this method along with an CE evaluator, a ResponseBuilder and a reference to a string. If the DDS has already been cached, the cached copy is read and returned (complete with the attributes and data) and the 'cache_token' value-result parameter is set to a string that should be used with the unlock_and_close() method to release the cache's lock on the response. If the DDS is not in the cache, the response is built, cached, locked and returned. In both cases, the cache-token is used to unlock the entry.

Note
The method read_dataset() looks in the cache and returns a valid entry if it's found. This method first looks at the cache because another process might have added an entry in the time between the two calls.
Todo:
This code is designed just for server function result caching. find a way to expand it to be general caching software for arbitrary DAP responses.
Todo:
This code utilizes an "unsafe" unlocking scheme in which it depends on the calling method to unlock the cache file. We should build an encapsulating version of, say, the DDS (called CachedDDS?) that carries the lock information and that will unlock the underlying cache file when destroyed (along with destroying the DDS of course).
Returns
The DDS that resulted from calling the server functions in the original CE.

Definition at line 432 of file BESStoredDapResultCache.cc.

References BESDEBUG, BESFileLockingCache::cache_too_big(), BESFileLockingCache::create_and_lock(), CRLF, BESDapResponseBuilder::dataset_constraint_ddx(), BESFileLockingCache::exclusive_to_shared_lock(), get_cache_file_name(), BESDapResponseBuilder::get_dataset_name(), BESFileLockingCache::get_read_lock(), BESFileLockingCache::purge_file(), BESFileLockingCache::unlock_cache(), BESFileLockingCache::update_and_purge(), and BESFileLockingCache::update_cache_info().

Here is the call graph for this function:

bool BESFileLockingCache::cache_too_big ( unsigned long long  current_size) const
virtualinherited

look at the cache size; is it too large? Look at the cache size and see if it is too big.

Returns
True if the size is too big, false otherwise.

Definition at line 660 of file BESFileLockingCache.cc.

Referenced by BESDapResponseCache::cache_dataset(), cache_dataset(), and BESFileLockingCache::update_and_purge().

bool BESFileLockingCache::create_and_lock ( const string &  target,
int &  fd 
)
virtualinherited

Create a file in the cache and lock it for write access.

If the file does not exist, make it, open it for read-write access and get an exclusive lock on it. The locking operation blocks, although that should never happen.

Parameters
targetThe name of the file to make/open/lock
fdValue-result param that holds the file descriptor of the opened file
Returns
True if the operation was successful, false otherwise. This method will return false if the file already existed (the file won't be locked and the descriptor reference is undefined - but likely -1).
Exceptions
BESBESInternalErrororif any error except EEXIST is returned by open(2) or if fcntl(2) returns an error.

Definition at line 481 of file BESFileLockingCache.cc.

References BESDEBUG, BESFileLockingCache::lock_cache_write(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), and cache_dataset().

Here is the call graph for this function:

void BESFileLockingCache::dump ( ostream &  strm) const
virtualinherited

dumps information about this object

Displays the pointer value of this instance along with information about this cache.

Parameters
strmC++ i/o stream to dump the information to

Implements BESObj.

Definition at line 898 of file BESFileLockingCache.cc.

References BESIndent::Indent(), BESIndent::LMarg(), and BESIndent::UnIndent().

Here is the call graph for this function:

void BESFileLockingCache::exclusive_to_shared_lock ( int  fd)
virtualinherited

Transfer from an exclusive lock to a shared lock.

If the file has an exclusive write lock on it, change that to a shared read lock. This is an atomic operation. If the call to fcntl(2) is protected by locking the cache, a dead lock will result given typical use of this class. This method exists to help with the situation where one process has the cache locked and is blocking on a shared read lock for a file that a second process has locked exclusively (for writing). By changing the exclusive lock to a shared lock, the first process can get its shared lock and then release the cache.

Parameters
fdThe file descriptor that is exclusively locked and which, on exit, will have a shared lock.

Definition at line 511 of file BESFileLockingCache.cc.

Referenced by BESDapResponseCache::cache_dataset(), and cache_dataset().

string BESStoredDapResultCache::get_cache_file_name ( const string &  src,
bool  mangle = false 
)
virtual

Build the name of file that will holds the uncompressed data from 'src' in the cache.

Note
How names are mangled: 'src' is the full name of the file to be cached.Tthe file name passed has an extension on the end that will be stripped once the file is cached. For example, if the full path to the file name is /usr/lib/data/fnoc1.nc.gz then the resulting file name will be #<prefix>#usr#lib#data#fnoc1.nc.
Parameters
srcThe source name to cache
mangleif True, assume the name is a file pathname and mangle it. If false, do not mangle the name (assume the caller has sent a suitable string) but do turn the string into a pathname located in the cache directory with the cache prefix. the 'mangle' param is true by default.

Reimplemented from BESFileLockingCache.

Definition at line 571 of file BESStoredDapResultCache.cc.

References BESDEBUG, BESFileLockingCache::getCacheDirectory(), and BESFileLockingCache::getCacheFilePrefix().

Referenced by cache_dataset().

Here is the call graph for this function:

unsigned long long BESFileLockingCache::get_cache_size ( )
virtualinherited

Get the cache size.

Read the size information from the cache info file and return it. This methods locks the cache.

Returns
The size of the cache.

Definition at line 672 of file BESFileLockingCache.cc.

References BESFileLockingCache::lock_cache_read(), and BESFileLockingCache::unlock_cache().

Referenced by BESFileLockingCache::purge_file().

Here is the call graph for this function:

BESStoredDapResultCache * BESStoredDapResultCache::get_instance ( const string &  cache_dir,
const string &  prefix,
unsigned long long  size 
)
static

Get an instance of the BESStoreResultCache object.

This class is a singleton, so the first call to any of three 'get_instance()' methods makes an instance and subsequent calls return a pointer to that instance.

Parameters
cache_dir_keyKey to use to get the value of the cache directory
prefix_keyKey for the item/file prefix. Each file added to the cache uses this as a prefix so cached items can be easily identified when /tmp is used for the cache.
size_keyHow big should the cache be, in megabytes
Returns
A pointer to a BESStoreResultCache object

Definition at line 215 of file BESStoredDapResultCache.cc.

References BESDEBUG, and BESError::get_message().

Here is the call graph for this function:

BESStoredDapResultCache * BESStoredDapResultCache::get_instance ( )
static

Get the default instance of the BESStoreResultCache object.

This will read "TheBESKeys" looking for the values of FUNCTION_CACHE_PATH, FUNCTION_CACHE_PREFIX, an FUNCTION_CACHE_SIZE to initialize the cache.

Definition at line 234 of file BESStoredDapResultCache.cc.

References BESDEBUG, BESError::get_message(), and getStoredResultsDirFromConfig().

Here is the call graph for this function:

bool BESFileLockingCache::get_read_lock ( const string &  target,
int &  fd 
)
virtualinherited

Get a read-only lock on the file if it exists.

Try to get a read-only lock on the file, blocking until we can get it. If the file does not exist, return false.

Note
If this code returns false, that means the file did not exist in the cache at the time of the test. by the time the caller gets the result, the file may have been added to the cache by another process.
Parameters
srcsrc file that will be cached eventually
targeta value-result parameter set to the resulting cached file
Returns
true if the file is in the cache and has been locked, false if the file is/was not in the cache.
Exceptions
Errorif the attempt to get the (shared) lock failed for any reason other than that the file does/did not exist.

Definition at line 453 of file BESFileLockingCache.cc.

References BESDEBUG, BESFileLockingCache::lock_cache_read(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), and cache_dataset().

Here is the call graph for this function:

const string BESFileLockingCache::getCacheDirectory ( )
inherited

Definition at line 885 of file BESFileLockingCache.cc.

Referenced by get_cache_file_name().

const string BESFileLockingCache::getCacheFilePrefix ( )
inherited

Definition at line 881 of file BESFileLockingCache.cc.

Referenced by get_cache_file_name().

unsigned long BESStoredDapResultCache::getCacheSizeFromConfig ( )
static

Definition at line 79 of file BESStoredDapResultCache.cc.

References BESDEBUG, BESKeys::get_value(), and TheBESKeys::TheKeys().

Here is the call graph for this function:

string BESStoredDapResultCache::getResultPrefixFromConfig ( )
static

Definition at line 119 of file BESStoredDapResultCache.cc.

References BESDEBUG, BESKeys::get_value(), BESUtil::lowercase(), and TheBESKeys::TheKeys().

Here is the call graph for this function:

string BESStoredDapResultCache::getStoredResultsDirFromConfig ( )
static

Definition at line 135 of file BESStoredDapResultCache.cc.

References BES_CATALOG_ROOT, BES_DATA_ROOT, BESDEBUG, BESKeys::get_value(), and TheBESKeys::TheKeys().

Referenced by get_instance().

Here is the call graph for this function:

string BESStoredDapResultCache::getSubDirFromConfig ( )
static

Definition at line 97 of file BESStoredDapResultCache.cc.

References BESDEBUG, BESKeys::get_value(), and TheBESKeys::TheKeys().

Here is the call graph for this function:

void BESFileLockingCache::initialize ( const string &  cache_dir,
const string &  prefix,
unsigned long long  size 
)
protectedinherited

Definition at line 88 of file BESFileLockingCache.cc.

void BESFileLockingCache::lock_cache_read ( )
virtualinherited

Get a shared lock on the 'cache info' file.

Definition at line 545 of file BESFileLockingCache.cc.

References BESDEBUG.

Referenced by BESFileLockingCache::get_cache_size(), and BESFileLockingCache::get_read_lock().

void BESFileLockingCache::lock_cache_write ( )
virtualinherited

Get an exclusive lock on the 'cache info' file.

The 'cache info' file is used to control certain cache actions, ensuring that they are atomic. These include making sure that the create_and_lock() and read_and_lock() operations are atomic as well as the purge and related operations.

Note
This is intended to be used internally only but might be useful in some settings.

Definition at line 533 of file BESFileLockingCache.cc.

References BESDEBUG.

Referenced by BESFileLockingCache::create_and_lock(), BESFileLockingCache::purge_file(), BESFileLockingCache::update_and_purge(), and BESFileLockingCache::update_cache_info().

void BESFileLockingCache::purge_file ( const string &  file)
virtualinherited

Purge a single file from the cache.

Purge a single file from the cache. The file might be old, etc., and need to be removed. Don't use this to shrink the cache when it gets too big, use update_and_purge() instead since that file optimizes accesses to the cache control file for several changes in a row.

Todo:
This is a new feature; add to BESCache3
Parameters
fileThe name of the file to purge.

Definition at line 839 of file BESFileLockingCache.cc.

References BESDEBUG, BESFileLockingCache::get_cache_size(), BESFileLockingCache::lock_cache_write(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), and cache_dataset().

Here is the call graph for this function:

void BESFileLockingCache::unlock_and_close ( const string &  file_name)
virtualinherited

Unlock the named file.

This does not do any name mangling; it just closes and unlocks whatever is named (or throws BESBESInternalErroror if the file cannot be closed). If the file was opened more than once, all descriptors are closed. If you need to close a specific descriptor, use the other version of unlock_and_close().

Note
This method assumes that the file was opend/locked using one of read_and_lock() or create_and_lock(). Those methods record the name/file- descriptor pairs so that the files can be properly closed and locks released.
Parameters
file_nameThe name of the file to unlock.
Exceptions
BESBESInternalErroror

Definition at line 583 of file BESFileLockingCache.cc.

References BESDEBUG.

void BESFileLockingCache::unlock_and_close ( int  fd)
virtualinherited

Unlock the file.

See also
unlock_and_close(const string &)
Parameters
fdThe descriptor of the file to unlock.
Exceptions
BESBESInternalErroror

Definition at line 599 of file BESFileLockingCache.cc.

References BESDEBUG.

void BESFileLockingCache::unlock_cache ( )
virtualinherited
void BESFileLockingCache::update_and_purge ( const string &  new_file)
virtualinherited

Purge files from the cache.

Purge files, oldest to newest, if the current size of the cache exceeds the size of the cache specified in the constructor. This method uses an exclusive lock on the cache for the duration of the purge process.

Parameters
new_fileThe name of a file this process just added to the cache. Using fcntl(2) locking there is no way this process can detect its own lock, so the shared read lock on the new file won't keep this process from deleting it (but will keep other processes from deleting it).

Definition at line 755 of file BESFileLockingCache.cc.

References BESDEBUG, BESISDEBUG, BESFileLockingCache::cache_too_big(), BESFileLockingCache::lock_cache_write(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), and cache_dataset().

Here is the call graph for this function:

unsigned long long BESFileLockingCache::update_cache_info ( const string &  target)
virtualinherited

Update the cache info file to include 'target'.

Add the size of the named file to the total cache size recorded in the cache info file. The cache info file is exclusively locked by this method for its duration. This updates the cache info file and returns the new size.

Parameters
targetThe name of the file
Returns
The new size of the cache

Definition at line 618 of file BESFileLockingCache.cc.

References BESDEBUG, BESFileLockingCache::lock_cache_write(), and BESFileLockingCache::unlock_cache().

Referenced by BESDapResponseCache::cache_dataset(), and cache_dataset().

Here is the call graph for this function:

Friends And Related Function Documentation

friend class StoredResultTest
friend

Definition at line 64 of file BESStoredDapResultCache.h.

Member Data Documentation

const string BESStoredDapResultCache::PREFIX_KEY = "DAP.StoredResultsCache.prefix"
static

Definition at line 77 of file BESStoredDapResultCache.h.

const string BESStoredDapResultCache::SIZE_KEY = "DAP.StoredResultsCache.size"
static

Definition at line 78 of file BESStoredDapResultCache.h.

const string BESStoredDapResultCache::SUBDIR_KEY = "DAP.StoredResultsCache.subdir"
static

Definition at line 76 of file BESStoredDapResultCache.h.


The documentation for this class was generated from the following files: