com.sleepycat.je.cleaner
Class Cleaner

java.lang.Object
  com.sleepycat.je.cleaner.Cleaner

All Implemented Interfaces:

EnvConfigObserver, DaemonRunner

public class Cleaner
extends java.lang.Object
implements DaemonRunner, EnvConfigObserver

The Cleaner is responsible for effectively garbage collecting the JE log. It looks through log files and locates log records (IN's and LN's of all flavors) that are superceded by later versions. Those that are "current" are propagated to a newer log file so that older log files can be deleted.

Field Summary

(package private) static java.lang.String	CLEAN_IN
(package private) static java.lang.String	CLEAN_LN
(package private) static java.lang.String	CLEAN_MIGRATE_LN
(package private) static java.lang.String	CLEAN_PENDING_LN
(package private) long	cleanerBytesInterval
(package private) boolean	clusterAll
(package private) boolean	clusterResident
(package private) java.util.logging.Level	detailedTraceLevel
(package private) static boolean	DO_CRITICAL_EVICTION Whether the cleaner should participate in critical eviction.
(package private) boolean	expunge
(package private) boolean	fetchObsoleteSize
(package private) boolean	lazyMigration
(package private) long	lockTimeout
(package private) int	lookAheadCacheSize
(package private) int	maxBatchFiles
(package private) int	nBacklogFiles
(package private) long	nCleanerDeletions
(package private) long	nCleanerRuns
(package private) long	nClusterLNsProcessed
(package private) long	nDeadlockRetries
(package private) long	nEntriesRead
(package private) long	nINsCleaned
(package private) long	nINsDead
(package private) long	nINsMigrated
(package private) long	nINsObsolete
(package private) long	nLNQueueHits
(package private) long	nLNsCleaned
(package private) long	nLNsDead
(package private) long	nLNsLocked
(package private) long	nLNsMarked
(package private) long	nLNsMigrated
(package private) long	nLNsObsolete
(package private) long	nMarkedLNsProcessed
(package private) long	nPendingLNsLocked
(package private) long	nPendingLNsProcessed
(package private) long	nRepeatIteratorReads
(package private) long	nToBeCleanedLNsProcessed
(package private) static boolean	PROACTIVE_MIGRATION Whether to fetch LNs for files in the to-be-cleaned set during lazy migration.
(package private) int	readBufferSize
(package private) boolean	trackDetail
(package private) static CacheMode	UPDATE_GENERATION Whether to update the IN generation count during searches.

Constructor Summary

Cleaner(EnvironmentImpl env, java.lang.String name)

Method Summary

(package private) void	addPendingDB(DatabaseImpl db) Adds the DB ID to the pending DB set if it is being deleted but deletion is not yet complete.
void	clearDeleteProhibited()
void	close() Release resources and update memory budget.
(package private) void	deleteSafeToDeleteFiles() Deletes all files that are safe-to-delete, if there are no read/only processes and concurrent backups.
int	doClean(boolean cleanMultipleFiles, boolean forceCleaning) Cleans selected files and returns the number of files cleaned.
void	envConfigUpdate(DbConfigManager cm, EnvironmentMutableConfig ignore) Process notifications of mutable property changes.
boolean	getFetchObsoleteSize()
FileSelector.CheckpointStartCleanerState	getFilesAtCheckpointStart() Returns a copy of the cleaned and processed files at the time a checkpoint starts.
(package private) byte[]	getLNMainKey(BIN bin, int index) Returns the main key for a given BIN entry.
int	getNWakeupRequests()
UtilizationProfile	getUtilizationProfile()
UtilizationTracker	getUtilizationTracker()
(package private) void	injectFileForCleaning(java.lang.Long fileNum) For unit testing.
boolean	isEvictable(BIN bin, int index) Returns whether the given BIN entry may be stripped by the evictor.
void	lazyMigrateDupCountLN(DIN din, ChildReference dclRef, boolean proactiveMigration) This method should be called just before logging a root DIN.
void	lazyMigrateLNs(BIN bin, boolean proactiveMigration, boolean backgroundIO) This method should be called just before logging a BIN.
void	loadStats(StatsConfig config, EnvironmentStats stat) Load stats.
(package private) void	processPending() If any LNs are pending, process them.
void	requestShutdown()
void	runOrPause(boolean run)
void	setDeleteProhibited()
void	setExceptionListener(ExceptionListener exceptionListener)
void	shutdown()
(package private) void	trace(java.util.logging.Level level, java.lang.String action, Node node, long logLsn, boolean completed, boolean obsolete, boolean dirtiedMigrated) Send trace messages to the java.util.logger.
void	updateFilesAtCheckpointEnd(FileSelector.CheckpointStartCleanerState info) When a checkpoint is complete, update the files that were returned at the beginning of the checkpoint.
void	updateReadOnlyFileCollections() Update the lowUtilizationFiles and mustBeCleanedFiles fields with new read-only collections, and update the backlog file count.
void	wakeup()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLEAN_IN

static final java.lang.String CLEAN_IN

See Also:

Constant Field Values

CLEAN_LN

static final java.lang.String CLEAN_LN

See Also:

Constant Field Values

CLEAN_MIGRATE_LN

static final java.lang.String CLEAN_MIGRATE_LN

See Also:

Constant Field Values

CLEAN_PENDING_LN

static final java.lang.String CLEAN_PENDING_LN

See Also:

Constant Field Values

PROACTIVE_MIGRATION

static final boolean PROACTIVE_MIGRATION

Whether to fetch LNs for files in the to-be-cleaned set during lazy migration. This is currently enabled because we do not support the dynamic addition of cleaner threads; that way, if the configured cleaner threads cannot keep up, we use proactive migration to keep up.

See Also:

Constant Field Values

UPDATE_GENERATION

static final CacheMode UPDATE_GENERATION

Whether to update the IN generation count during searches. This is currently disabled because 1) we update the generation of the BIN when we set a MIGRATE flag and 2) if the BIN is not evicted its parents will not be, so not updating the generation during the search has no benefit. By not updating the generation during searches for which we do NOT set the MIGRATE flag, we avoid holding INs in the cache that are not needed for lazy migration. However, we do very few searches for obsolete LNs because the obsolete tracking info prevents this, so the benefit of not updating the generation during searches is questionable. In other words, changing this setting will have little effect.

DO_CRITICAL_EVICTION

static final boolean DO_CRITICAL_EVICTION

Whether the cleaner should participate in critical eviction. Ideally the cleaner would not participate in eviction, since that would reduce the cost of cleaning. However, the cleaner can add large numbers of nodes to the cache. By not participating in eviction, other threads could be kept in a constant state of eviction and would effectively starve. Therefore, this setting is currently enabled.

See Also:

Constant Field Values

nBacklogFiles

int nBacklogFiles

nCleanerRuns

long nCleanerRuns

nCleanerDeletions

long nCleanerDeletions

nINsObsolete

long nINsObsolete

nINsCleaned

long nINsCleaned

nINsDead

long nINsDead

nINsMigrated

long nINsMigrated

nLNsObsolete

long nLNsObsolete

nLNsCleaned

long nLNsCleaned

nLNsDead

long nLNsDead

nLNsLocked

long nLNsLocked

nLNsMigrated

long nLNsMigrated

nLNsMarked

long nLNsMarked

nLNQueueHits

long nLNQueueHits

nPendingLNsProcessed

long nPendingLNsProcessed

nMarkedLNsProcessed

long nMarkedLNsProcessed

nToBeCleanedLNsProcessed

long nToBeCleanedLNsProcessed

nClusterLNsProcessed

long nClusterLNsProcessed

nPendingLNsLocked

long nPendingLNsLocked

nEntriesRead

long nEntriesRead

nRepeatIteratorReads

long nRepeatIteratorReads

lockTimeout

long lockTimeout

readBufferSize

int readBufferSize

lookAheadCacheSize

int lookAheadCacheSize

nDeadlockRetries

long nDeadlockRetries

expunge

boolean expunge

clusterResident

boolean clusterResident

clusterAll

boolean clusterAll

maxBatchFiles

int maxBatchFiles

detailedTraceLevel

java.util.logging.Level detailedTraceLevel

cleanerBytesInterval

long cleanerBytesInterval

trackDetail

boolean trackDetail

fetchObsoleteSize

boolean fetchObsoleteSize

lazyMigration

boolean lazyMigration

Constructor Detail

Cleaner

public Cleaner(EnvironmentImpl env,
               java.lang.String name)
        throws DatabaseException

Throws:

DatabaseException

Method Detail

envConfigUpdate

public void envConfigUpdate(DbConfigManager cm,
                            EnvironmentMutableConfig ignore)
                     throws DatabaseException

Process notifications of mutable property changes.

Specified by:

envConfigUpdate in interface EnvConfigObserver

Throws:

DatabaseException

getUtilizationTracker

public UtilizationTracker getUtilizationTracker()

getUtilizationProfile

public UtilizationProfile getUtilizationProfile()

getFetchObsoleteSize

public boolean getFetchObsoleteSize()

runOrPause

public void runOrPause(boolean run)

Specified by:

runOrPause in interface DaemonRunner

wakeup

public void wakeup()

requestShutdown

public void requestShutdown()

Specified by:

requestShutdown in interface DaemonRunner

shutdown

public void shutdown()

Specified by:

shutdown in interface DaemonRunner

getNWakeupRequests

public int getNWakeupRequests()

Specified by:

getNWakeupRequests in interface DaemonRunner

setExceptionListener

public void setExceptionListener(ExceptionListener exceptionListener)

Specified by:

setExceptionListener in interface DaemonRunner

doClean

public int doClean(boolean cleanMultipleFiles,
                   boolean forceCleaning)
            throws DatabaseException

Cleans selected files and returns the number of files cleaned. This method is not invoked by a deamon thread, it is programatically.

Parameters:

cleanMultipleFiles - is true to clean until we're under budget, or false to clean at most one file.

forceCleaning - is true to clean even if we're not under the utilization threshold.

Returns:

the number of files cleaned, not including files cleaned unsuccessfully.

Throws:

DatabaseException

loadStats

public void loadStats(StatsConfig config,
                      EnvironmentStats stat)
               throws DatabaseException

Load stats.

Throws:

DatabaseException

injectFileForCleaning

void injectFileForCleaning(java.lang.Long fileNum)

For unit testing.

deleteSafeToDeleteFiles

void deleteSafeToDeleteFiles()
                       throws DatabaseException

Deletes all files that are safe-to-delete, if there are no read/only processes and concurrent backups. Deletion is coordinated by the synchronization variable deleteFileLock AND by the deleteProhibited state variable. The reason that two different mechanisms are use is because file deletion must be prevented both inter and intra-process. File locks must be used for inter-process, and the state bit for intra-process. To guard against read/only processes, the would-be deleter tries to get an exclusive lock on the environment. This will not be possible if a read/only process exists. To guard against backup mode, the would-be deleter checks the deleteProhibited state. Backup and file deletion can only be carried out by a read/write process, so both activities are working in the same process. Note that file locks are not supported intra-process. The deleteProhibited state is used rather than a simple synchronization on deleteFileLock because the start/endBackup is controlled by the application, and the copying of log files can take an arbitrarily long time. Using synchronization on deleteFileLock would make it possible to lock out a cleaner thread for an unacceptable amount of time. The deleteProhibited state is also set while a replication node is sending files to initialize another size. More than one such initialization may be occuring at once, and a backup may also be occuring simultaneously, hence deleteProhibited may be greater than 1.

Throws:

DatabaseException

setDeleteProhibited

public void setDeleteProhibited()

clearDeleteProhibited

public void clearDeleteProhibited()

getFilesAtCheckpointStart

public FileSelector.CheckpointStartCleanerState getFilesAtCheckpointStart()
                                                                   throws DatabaseException

Returns a copy of the cleaned and processed files at the time a checkpoint starts.

If non-null is returned, the checkpoint should flush an extra level, and addCheckpointedFiles() should be called when the checkpoint is complete.

Throws:

DatabaseException

updateFilesAtCheckpointEnd

public void updateFilesAtCheckpointEnd(FileSelector.CheckpointStartCleanerState info)
                                throws DatabaseException

When a checkpoint is complete, update the files that were returned at the beginning of the checkpoint.

Throws:

DatabaseException

updateReadOnlyFileCollections

public void updateReadOnlyFileCollections()

Update the lowUtilizationFiles and mustBeCleanedFiles fields with new read-only collections, and update the backlog file count.

processPending

void processPending()
              throws DatabaseException

If any LNs are pending, process them. This method should be called often enough to prevent the pending LN set from growing too large.

Throws:

DatabaseException

isEvictable

public boolean isEvictable(BIN bin,
                           int index)

Returns whether the given BIN entry may be stripped by the evictor. True is always returned if the BIN is not dirty. False is returned if the BIN is dirty and the entry will be migrated soon. Note that the BIN may or may not be latched when this method is called. Returning the wrong answer is OK in that case (it will be called again later when latched), but an exception should not occur.

lazyMigrateLNs

public void lazyMigrateLNs(BIN bin,
                           boolean proactiveMigration,
                           boolean backgroundIO)
                    throws DatabaseException

This method should be called just before logging a BIN. LNs will be migrated if the MIGRATE flag is set, or if they are in a file to be cleaned, or if the LNs qualify according to the rules for cluster and clusterAll.

On return this method guarantees that no MIGRATE flag will be set on any child entry. If this method is *not* called before logging a BIN, then the addPendingLNs method must be called.

Parameters:

bin - is the latched BIN. The latch will not be released by this method.

proactiveMigration - perform proactive migration if needed; this is false during a split, to reduce the delay in the user operation.

Throws:

DatabaseException

lazyMigrateDupCountLN

public void lazyMigrateDupCountLN(DIN din,
                                  ChildReference dclRef,
                                  boolean proactiveMigration)
                           throws DatabaseException

This method should be called just before logging a root DIN. The DupCountLN will be migrated if the MIGRATE flag is set, or if it is in a file to be cleaned, or if the LN qualifies according to the rules for cluster and clusterAll.

On return this method guarantees that the MIGRATE flag will not be set on the child entry. If this method is *not* called before logging a root DIN, then the addPendingDupCountLN method must be called.

Parameters:

din - is the latched DIN. The latch will not be released by this method.

dclRef - is the reference to the DupCountLN.

proactiveMigration - perform proactive migration if needed; this is false during a split, to reduce the delay in the user operation.

Throws:

DatabaseException

getLNMainKey

byte[] getLNMainKey(BIN bin,
                    int index)
              throws DatabaseException

Returns the main key for a given BIN entry.

Throws:

DatabaseException

addPendingDB

void addPendingDB(DatabaseImpl db)

Adds the DB ID to the pending DB set if it is being deleted but deletion is not yet complete.

trace

void trace(java.util.logging.Level level,
           java.lang.String action,
           Node node,
           long logLsn,
           boolean completed,
           boolean obsolete,
           boolean dirtiedMigrated)

Send trace messages to the java.util.logger. Don't rely on the logger alone to conditionalize whether we send this message, we don't even want to construct the message if the level is not enabled.

close

public void close()

Release resources and update memory budget. Should only be called when this environment is closed and will never be accessed again.