Archive the paths associated with a single job. More...

#include <archiver.h>

Collaboration diagram for job_archiver:

Public Types
enum	archiving_status { status_pending, status_processing, status_reschedule, status_fatal_error, status_error, status_completed, status_done }

Public Member Functions
	job_archiver (const job *a_job)
	C'tor. More...

const std::string	prefix (void)
	Generate a job prefix string. More...

const std::string	id (void)
	Generate a job id string. More...

void	clear (void)
	Clear the job archiver and return it to it's initial state. More...

void	end (void)
	End any processes handling this job. More...

const archiving_status	status (void)
	Return the processing status of this job archiver. More...

void	start (void)
	Begin processing. More...

void	process (void)
	Parent processor for a job. More...

single_job_report	report (void) const
	Return the job report for this job. More...

Private Member Functions
void	mf_do_chores (void)
	Child processor for a job. More...

void	mf_process_report (const std::string &a_str)

void	mf_process_child_io (bool a_finalize)
	Process I/O from the child. More...

void	mf_process_rsync_io (execute &a_exec, uint16 a_timeout, uint64 &a_files_total, uint64 &a_files_xferd, uint64 &a_size_total, uint64 &a_size_xferd, bool &a_overflow_detected)
	Process I/O from rsync. More...

void	mf_parse_rsync_io (std::string a_str, uint64 &a_files_total, uint64 &a_files_xferd, uint64 &a_size_total, uint64 &a_size_xferd)
	Parse I/O from rsync. More...

void	mf_trim_string (std::string &a_str)
	Trim off all non-digit leading and trailing characters from a string. More...

void	mf_parse_report (const std::string &a_str)

Private Attributes
const job *	m_job

archiving_status	m_status

timer	m_timer

timer	m_io_timer

bool	m_rsync_timeout_flag

execute	m_exec

bool	m_success

estring	m_io_out

estring	m_io_err

pid_t	m_child_pid

job_path_report	m_jpr

single_job_report	m_jr

std::string	m_error_msg

Detailed Description

Archive the paths associated with a single job.

Definition at line 36 of file archiver.h.

Member Enumeration Documentation

enum job_archiver::archiving_status

Enumerator
status_pending
status_processing
status_reschedule
status_fatal_error
status_error
status_completed
status_done

Definition at line 38 of file archiver.h.

Constructor & Destructor Documentation

job_archiver::job_archiver ( const job * a_job )

C'tor.

Set a job to be assiciated with this job archiver and initialize it's processing status to "pending".

Definition at line 124 of file archiver.cc.

References clear(), m_job, m_status, and status_pending.

Here is the call graph for this function:

Member Function Documentation

void job_archiver::clear ( void )

Clear the job archiver and return it to it's initial state.

End any processes handling this job and return the job archiver to it's "pending" state.

Definition at line 166 of file archiver.cc.

References job_path_report::clear(), single_job_report::clear(), end(), m_child_pid, m_error_msg, m_exec, m_jpr, m_jr, m_rsync_timeout_flag, m_status, m_success, execute::my_pid(), and status_pending.

Referenced by job_archiver().

Here is the call graph for this function:

void job_archiver::end ( void )

End any processes handling this job.

If any child processes are handling this job, terminate them. Erase any pending I/O for the now defunct child. Set our processing status to "done".

Definition at line 184 of file archiver.cc.

References execute::child_running(), execute::clear(), execute::kill_child(), logger, configuration_manager::logging_manager, m_child_pid, m_exec, m_io_err, m_io_out, m_io_timer, m_status, m_timer, prefix(), status_done, timer::stop(), and log_manager::write().

Referenced by clear().

Here is the call graph for this function:

const std::string job_archiver::id ( void )

Generate a job id string.

Definition at line 149 of file archiver.cc.

References job::generate_job_id(), m_job, and prefix().

Here is the call graph for this function:

void job_archiver::mf_do_chores ( void )

private

Child processor for a job.

For each path in this job:

Create the directory heiararchy for this job in the archive
Until done or until out of retrys:
- Choose a hardlink source, if applicable and available
- Construct the command line to pass to rsync
- Spawn rsync
- Process I/O sent back from rsync
- Process exit code or signal number returned from rsync
Generate and submit a report to the report manager

Definition at line 406 of file archiver.cc.

References archiver, execute::child_exit_code(), execute::child_exited_normally(), execute::child_signal_no(), execute::child_signaled(), config, job::connection_local, job::connection_ssh_local, ERROR, job::excludes, execute::exec(), exists(), rstat::exit(), rsync_behavior::fail, job::generate_archive_path(), job::generate_rsync_options_vector(), job::generate_source_path(), job::generate_ssh_options_vector(), job::hostname, job::includes, is_timestamp(), logger, m_error_msg, m_io_timer, m_job, m_rsync_timeout_flag, m_success, mf_process_rsync_io(), mk_relative_path(), rsync_behavior::ok, subdirectory::path(), job::paths, reform_path(), relative_path(), rsync_behavior::retry_without_hardlinks, job::rsync_behavior, job::rsync_connection, rsync_estat_str, job::rsync_hardlink, configuration_manager::rsync_local_path(), job::rsync_multi_hardlink, job::rsync_multi_hardlink_max, job::rsync_remote_path, job::rsync_remote_user, job::rsync_retry_count, job::rsync_retry_delay, job::rsync_timeout, log_manager::set_error_logging(), rstat::signal(), configuration_manager::ssh_local_path(), timer::start(), timer::stop(), timestamp::str(), configuration_manager::timestamp(), TRY_nomem, vault_manager::vault(), vaulter, archive_manager::working_archive_path(), writable(), and reportio::write_report().

Referenced by start().

Here is the call graph for this function:

void job_archiver::mf_parse_report ( const std::string & a_str )

private

void job_archiver::mf_parse_rsync_io	(	std::string	a_str,
		uint64 &	a_files_total,
		uint64 &	a_files_xferd,
		uint64 &	a_size_total,
		uint64 &	a_size_xferd
	)

private

Parse I/O from rsync.

Search for special output from rsync to tell us something about the number and size of files and files transfered.

Definition at line 821 of file archiver.cc.

References err_unknown, ERROR_INSTANCE, mf_trim_string(), and error::push_back().

Referenced by mf_process_rsync_io().

Here is the call graph for this function:

void job_archiver::mf_process_child_io ( bool a_finalize )

private

Process I/O from the child.

While there is I/O to be read, read and parse it. When the end of a line is reached write that line to the log file. If a_finalize is true, the flush the child I/O buffer string.

Definition at line 730 of file archiver.cc.

References config, execute::err_eof(), execute::err_read(), execute::err_ready(), configuration_manager::io_poll_interval(), logger, configuration_manager::logging_rsync, m_child_pid, m_exec, m_io_err, m_io_out, mf_process_report(), execute::out_eof(), execute::out_read(), execute::out_ready(), prefix(), and log_manager::write().

Referenced by process().

Here is the call graph for this function:

void job_archiver::mf_process_report ( const std::string & a_str )

private

Definition at line 715 of file archiver.cc.

References single_job_report::add_report(), m_jpr, m_jr, and reportio::parse().

Referenced by mf_process_child_io().

Here is the call graph for this function:

void job_archiver::mf_process_rsync_io	(	execute &	a_exec,
		uint16	a_timeout,
		uint64 &	a_files_total,
		uint64 &	a_files_xferd,
		uint64 &	a_size_total,
		uint64 &	a_size_xferd,
		bool &	a_overflow_detected
	)

private

Process I/O from rsync.

If there is I/O from rsync to be read, read it and then send it through the parser.

Definition at line 943 of file archiver.cc.

References execute::child_running(), config, timer::duration_secs(), execute::err_fd(), execute::err_ready(), configuration_manager::io_poll_interval(), execute::kill_child(), m_error_msg, m_io_timer, m_rsync_timeout_flag, mf_parse_rsync_io(), execute::out_fd(), execute::out_ready(), vault_manager::overflow(), timer::start(), timer::stop(), vaulter, reportio::write_rsync_err(), and reportio::write_rsync_out().

Referenced by mf_do_chores().

Here is the call graph for this function:

void job_archiver::mf_trim_string ( std::string & a_str )

private

Trim off all non-digit leading and trailing characters from a string.

Definition at line 807 of file archiver.cc.

Referenced by mf_parse_rsync_io().

const std::string job_archiver::prefix ( void )

Generate a job prefix string.

Create a string to uniquely identify this job to be used in the log file to uniquely identify this job

Definition at line 137 of file archiver.cc.

References m_job.

Referenced by end(), id(), mf_process_child_io(), process(), and start().

void job_archiver::process ( void )

Parent processor for a job.

Check for I/O from the child. Check the child's status to see if it's still running, has exited with an exit code, or has exited from a signal. If the child sis not exit normally (i.e. exit from a signal or exit with a non-zero exit code) then check the vault for overflow. If the vault has exceeded it's overflow threshold then that could be the cause for the child's failure, in which case we reschedule the child to be processed again later.

If the job is finished (whether successful or not), update the job archiver's status to "completed".

Definition at line 327 of file archiver.cc.

References execute::child_exit_code(), execute::child_exited_normally(), execute::child_running(), execute::child_signal_no(), execute::child_signaled(), timer::duration(), logger, m_exec, m_io_timer, m_status, m_timer, mf_process_child_io(), prefix(), status_completed, status_error, timer::stop(), and log_manager::write().

Here is the call graph for this function:

single_job_report job_archiver::report ( void ) const

Return the job report for this job.

Definition at line 388 of file archiver.cc.

References m_jr.

void job_archiver::start ( void )

Begin processing.

Attempt to fork a child process to handle this job. If unsuccessful then retry again later. The child then calls mf_do_chores() to handle the actual processing, while the parent updates the job archiver's status from "pending" to "processing" and begins a timer to measure the duration of the job process.

Definition at line 217 of file archiver.cc.

References archiver, execute::child_pid(), err_unknown, exists(), execute::exit(), execute::fork(), job::generate_archive_path(), job::generate_job_id(), single_job_report::id(), execute::is_child(), logger, m_child_pid, m_exec, m_io_timer, m_job, m_jr, m_rsync_timeout_flag, m_status, m_success, m_timer, mf_do_chores(), mk_dirhier(), job::paths, prefix(), reform_path(), execute::reroute_stdio(), timer::start(), status_processing, status_reschedule, error::str(), archive_manager::working_archive_path(), and log_manager::write().