Object to open and manage a logging file, low-level API. More...

#include <FileHandler.hxx>

Inheritance diagram for FileHandler:

Collaboration diagram for FileHandler:

Classes
struct	read_job
	Read jobs, need requester and the offset. More...

struct	StreamSetInfo
	Collection of data per stream. More...

Public Types
enum class	Mode { New , Truncate , Append , Any , Read }
	Mode for opening the file. More...

typedef std::ios::off_type	pos_type
	Use the position type of ios streams.

typedef boost::intrusive_ptr< FileHandler >	pointer
	Pointer type to preferably use.

Public Member Functions
	FileHandler (const std::string &fname, Mode mode=Mode::Truncate, unsigned blocksize=4096U)
	Constructor for an object managing a ddff log file.

	FileHandler ()
	Default constructor, requires the open() call to be useful.

void	open (const std::string &fname, Mode mode=Mode::Truncate, unsigned blocksize=4096U)
	Open, after using a default constructor.

virtual	~FileHandler ()
	Destructor.

FileStreamWrite::pointer	createWrite (size_t bufsize=0)
	Create a new write stream.

FileStreamWrite::pointer	attachWrite (unsigned sid, size_t bufsize=0)
	Attach to an existing write stream, only possible if that stream has no writer (i.e., after analysis of an existing file)

unsigned	processWrites ()
	Write available buffers to file.

virtual void	syncToFile (bool intermediate=true)
	Perform an intermediate flush.

bool	isExtending ()
	Return true if the file existed before.

bool	isComplete () const
	check that the filer is correctly configured.

unsigned	getBlockSize () const
	return the blocksize used in this file

FileStreamRead::pointer	createRead (unsigned stream_id, unsigned num_cache=3U, bool slice_indexed=false)
	Create a read stream.

void	runLoads ()
	Force a load in the calling thread.

void	checkIndices (pos_type offset=0)
	Check the indices in the file.

virtual void	bufferWriteInformation (pos_type offset, DDFFMessageBuffer::ptr_type buffer)
	Callback, informing about where a block/buffer is written.

void	requestFileStreamReadRelease (FileStreamRead::pointer &ptr)
	Inform of destruction of a filestreamRead object.

Static Public Attributes
static constexpr std::ios::off_type	pos_type_max
	Max pos type is used to mark "infinite"/undefined.

Protected Types
typedef std::vector< StreamSetInfo >	streams_t
	List of issued writers and readers.

Protected Attributes
unsigned	blocksize
	Gross size of file blocks.

streams_t	streams
	List of issued writers.

AsyncList< FileStreamWrite * >	write_jobs
	FIFO with writing work to do.

AsyncList< read_job >	read_jobs
	FIFO with loading work to do.

pos_type	offset
	Give current write offset in the file.

std::string	filename
	File name, to remember.

std::fstream	file
	File stream.

Mode	open_mode
	Open mode.

bool	file_existing
	Is this a new file ?

Friends
class	FileStreamWrite
	Calls for fileStreamWrite.

class	FileStreamRead
	Calls for fileStreamRead.

Detailed Description

Object to open and manage a logging file, low-level API.

This handles a file in the DelftDataFormat file format, which can contain multiple data streams. A "stream" is implemented as a series of linked blocks in this file.

This class can return a writer for a new stream, or readers for a given stream.

File format

Block sizes are a multiple of a base block size, default 4096 bytes, and for best performance chosen to match (be a multiple of) physical device block size.

Each block has a 28-byte header, indicating next block location, block size, fill level, block number, stream id, checksum, etc. For the precise format see the description in ControlBlock.hxx. The remainder of the block contains data.

Reading a stream

For a stream, the starting block for reading is loaded with a FileStreamRead object. This object needs the offset of the first (or a later) block in the stream.

A FileStreamRead object can issue an iterator, initially using the first loaded block of data (excluding the header). When the iterator reaches the end of the data in the loaded block, the next block, if available, will be loaded and used. To improve real-time performance, it is also possible to pre-load blocks of data.

Before or after opening an existing file, create all needed read streams, and run checkIndices.

Writing a stream

A FileStreamWrite object can write a stream. It can issue an iterator, and will request file blocks from the FileHandler when needed.

Alternating reading and writing.

A file can be used for both reading and writing. After writing, an intermediate end can be written to a file by calling the syncToFile() function. A following call to runLoads() re-loads the necessary data into the FileStreamRead objects, which can then be used to read data.

Further interface.

A further elaboration of the interface is given in the dueca::ddff::FileWithInventory class. Here the #0 stream is reserved for information on named data streams, and the #1 stream is reserved for recording block offsets at points where stream reading may be started.

Error robustness and recovery

All blocks carry a checksum, calculated over the data and metadata (with the exception of the next block location). The metadata also contains (if so notified by the application) the offset for the first completely written written object in a block, where "object" can be defined by the application. This should enable partial recovery of data in the case of data corruption.

Member Enumeration Documentation

◆ Mode

enum class FileHandler::Mode

strong

Mode for opening the file.

Enumerator
New	Only open a file as new, there may be no existing file with this name.
Truncate	Open a new file, or overwrite the old, discard any existing data.
Append	Open a new or existing file, read and prepare to extend if existing.
Any	Open an existing file to append, or a new one.
Read	Open an existing file read-only.

Constructor & Destructor Documentation

◆ FileHandler()

FileHandler::FileHandler	(	const std::string &	fname,
		Mode	mode = Mode::Truncate,
		unsigned	blocksize = 4096U )

Constructor for an object managing a ddff log file.

Parameters

fname	Name for the file
mode	Open mode, see FileHandler::Mode
blocksize	File block size. It is advised to let this match your platform's physical block size. Note that write streams can use a buffer equal to or a multiple of the block size chosen here.

◆ ~FileHandler()

virtual FileHandler::~FileHandler ( )

virtual

Destructor.

This will write any remaining blocks and close the file

Member Function Documentation

◆ createWrite()

FileStreamWrite::pointer FileHandler::createWrite ( size_t bufsize = 0 )

Create a new write stream.

Parameters

bufsize Size of the buffers in this stream. Should be a multiple of the handler's blocksize.

Returns: Pointer to the write object.

◆ attachWrite()

FileStreamWrite::pointer FileHandler::attachWrite	(	unsigned	sid,
		size_t	bufsize = 0 )

Attach to an existing write stream, only possible if that stream has no writer (i.e., after analysis of an existing file)

Parameters

sid	Stream ID.
bufsize	Requested size for buffers, takes default otherwise.

Returns: Pointer to the write object.

◆ processWrites()

unsigned FileHandler::processWrites ( )

Write available buffers to file.

Buffers that have been offered to the handler as complete are written to file at this call.

Returns: number of processed writes

◆ syncToFile()

virtual void FileHandler::syncToFile ( bool intermediate = true )

virtual

Perform an intermediate flush.

All writers are checked for partial buffers, these are pushed to for writing and then written to file. Further writing in these buffers is possible when intermediate is set to true.

Parameters

intermediate Perform an intermediate flush; after this further writes are still possible.

◆ isComplete()

bool FileHandler::isComplete ( ) const

check that the filer is correctly configured.

Returns: True, if opened.

◆ createRead()

FileStreamRead::pointer FileHandler::createRead	(	unsigned	stream_id,
		unsigned	num_cache = 3U,
		bool	slice_indexed = false )

Create a read stream.

Note that each stream_id can only have one read object. The object is only initialised after a checkIndices() call, at which point the file is read and information on the blocks for this stream is provided.

Parameters

stream_id	Given id.
num_cache	Preference for the number of cached blocks.
slice_indexed	Indexing is per data slice, the stream stores segments or slices of this data type.

Returns: A

◆ runLoads()

void FileHandler::runLoads ( )

Force a load in the calling thread.

Note that this may block and use a lock. For real-time operation, loads should happen on request, and with enough lead time so that needed data is available in buffers. If a read from a stream is attempted and the data is not loaded, the runLoads() call will be called upon the read attempt.

◆ checkIndices()

void FileHandler::checkIndices ( pos_type offset = 0 )

Check the indices in the file.

This runs through an opened file, and checks block sizes and stream ID's. Any reported FileStreamRead objects matching the ID's will get notified of the file offsets of the blocks.

◆ requestFileStreamReadRelease()

void FileHandler::requestFileStreamReadRelease ( FileStreamRead::pointer & ptr )

Inform of destruction of a filestreamRead object.

Note that after this call the FileStreamRead object is no longer effective, and should be released.

Member Data Documentation

◆ pos_type_max

constexpr std::ios::off_type FileHandler::pos_type_max

staticconstexpr

Initial value:

=

std::numeric_limits<std::ios::off_type>::max()

Max pos type is used to mark "infinite"/undefined.

The documentation for this class was generated from the following file:

/home/abuild/rpmbuild/BUILD/dueca-4.1.3/ddff/FileHandler.hxx

Classes

Public Types

Public Member Functions

Static Public Attributes

Protected Types

Protected Attributes

Friends

Detailed Description

File format

Reading a stream

Writing a stream

Alternating reading and writing.

Further interface.

Error robustness and recovery

Member Enumeration Documentation

◆ Mode

Constructor & Destructor Documentation

◆ FileHandler()

◆ ~FileHandler()

Member Function Documentation

◆ createWrite()

◆ attachWrite()

◆ processWrites()

◆ syncToFile()

◆ isComplete()

◆ createRead()

◆ runLoads()

◆ checkIndices()

◆ requestFileStreamReadRelease()

Member Data Documentation

◆ pos_type_max