freenet.client.async

Class SplitFileFetcher

java.lang.Object

freenet.client.async.SplitFileFetcher

All Implemented Interfaces:

ClientGetState, SplitFileFetcherStorageCallback, java.io.Serializable

public class SplitFileFetcher
extends java.lang.Object
implements ClientGetState, SplitFileFetcherStorageCallback, java.io.Serializable

Splitfile fetcher based on keeping as much state as possible, and in particular the downloaded blocks, in a single file. The main goals here are: 1) Minimising disk seeks. E.g. in the older versions we abstracted out block storage, this caused a lot of unnecessary seeking and copying. It's better to keep the downloaded data close together on disk. 2) Robustness. This should be robust even against moderate levels of on-disk data corruption. And it's separate for each splitfile. And it has fixed disk usage, until it completes. Some of this robustness violates layering e.g. checking blocks against the CHKs they are supposed to represent. This actually simplifies matters e.g. when decoding a segment, and allows us to not only recover from almost any error (at least in the parts that change, which are most likely to be corrupted), but also to do so efficiently. The SplitFileFetcher*Storage classes manage storing the data and FEC decoding. This class deals with everything else: The interface to the rest of the client layer, selection of keys to fetch, listening for blocks, etc. PERSISTENCE ROADMAP: Now: Currently this class does not support persistence. Near future goal: Support persistence within the database. Stay in memory, do not deactivate. Store this class in the database but not its *Storage classes. Load the SplitFileFetcherStorage on loading Freenet, resuming existing downloads. => Significant improvement in reliability and disk I/O. In principle we could resume downloads separately from the database e.g. if it breaks. Too complicated in practice because of other data structures. - Require: Add "persistence", hashCode, force no deactivation, activate where necessary (when dealing with other stuff). Fill in context and load storage when activated on startup. Longer term goal: Implement similar code for inserts. Eliminate db4o and persist everything that remains with simple checkpointed serialisation. LOCKING: (this) should be taken last, because it is used by e.g. SplitFileFetcherGet.isCancelled().

See Also:

Serialized Form

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	SplitFileFetcher()
	SplitFileFetcher(ClientGetter getter, java.io.DataInputStream dis, ClientContext context)
	SplitFileFetcher(Metadata metadata, GetCompletionCallback rcb, ClientRequester parent, FetchContext fetchContext, boolean realTimeFlag, java.util.List<Compressor.COMPRESSOR_TYPE> decompressors, ClientMetadata clientMetadata, long token, boolean topDontCompress, short topCompatibilityMode, boolean persistent, FreenetURI thisKey, boolean isFinalFetch, ClientContext context)

Method Summary

Modifier and Type	Method and Description
void	cancel(ClientContext context) Cancel the request, and call onFailure() on the callback in order to tell downstream (ultimately the client) that cancel has succeeded, and to allow it to call removeFrom() to avoid a database leak.
void	clearCooldown() Called when the fetcher may have exited cooldown early.
void	fail(FetchException e) Called when the fetch failed, e.g.
void	failOnDiskError(ChecksumFailedException e) Fail the whole splitfile request when we get unrecoverable data corruption, e.g.
void	failOnDiskError(java.io.IOException e) Fail the whole splitfile request when we get an IOException on writing to or reading from the on-disk storage.
HasKeyListener	getHasKeyListener() Can be null.
short	getPriorityClass() Get the priority class of the request.
KeySalter	getSalter()
BaseSendableGet	getSendableGet() Can be null.
long	getToken() Get a long value which may be passed around to identify this request (e.g.
boolean	hasFinished()
boolean	localRequestOnly()
void	maybeAddToBinaryBlob(ClientCHKBlock block) Called whenever we successfully download, decode or encode a block and it matches the expected key.
void	onClosed() Called when the storage layer has finished, the higher level code has finished, and the storage has been freed, i.e.
void	onFailedBlock() Called when the splitfile fetcher gives up on a block.
void	onFetchedBlock()
void	onResume(ClientContext context) Called on restarting the node for a persistent request.
void	onResume(int succeededBlocks, int failedBlocks, ClientMetadata meta, long finalSize)
void	onShutdown(ClientContext context) Called just before the final write of client.dat before the node shuts down.
void	onSplitfileCompatibilityMode(InsertContext.CompatibilityMode min, InsertContext.CompatibilityMode max, byte[] customSplitfileKey, boolean compressed, boolean bottomLayer, boolean definitiveAnyway) Called during construction, when we know the settings for the splitfile.
void	onSuccess() The splitfile download succeeded.
void	queueHeal(byte[] data, byte[] cryptoKey, byte cryptoAlgorithm) Queue a block to be healed.
void	reduceCooldown(long wakeupTime) Called when the wakeup time reduces but it is still not fetchable.
void	restartedAfterDataCorruption() Called when we recover from disk corruption, and have to re-download some blocks that we had already downloaded but which were corrupted on disk.
void	schedule(ClientContext context) Schedule the request on the ClientRequestScheduler.
void	setSplitfileBlocks(int requiredBlocks, int remainingBlocks) Called during construction to tell other layers how many blocks to expect.
void	toNetwork()
boolean	wantBinaryBlob() Do we want maybeAddToBinaryBlob() to be called?? LOCKING: Should not take any locks.
boolean	writeTrivialProgress(java.io.DataOutputStream dos)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SplitFileFetcher

public SplitFileFetcher(Metadata metadata,
                        GetCompletionCallback rcb,
                        ClientRequester parent,
                        FetchContext fetchContext,
                        boolean realTimeFlag,
                        java.util.List<Compressor.COMPRESSOR_TYPE> decompressors,
                        ClientMetadata clientMetadata,
                        long token,
                        boolean topDontCompress,
                        short topCompatibilityMode,
                        boolean persistent,
                        FreenetURI thisKey,
                        boolean isFinalFetch,
                        ClientContext context)
                 throws FetchException,
                        MetadataParseException

Throws:

FetchException

MetadataParseException

SplitFileFetcher

protected SplitFileFetcher()

SplitFileFetcher

public SplitFileFetcher(ClientGetter getter,
                        java.io.DataInputStream dis,
                        ClientContext context)
                 throws StorageFormatException,
                        ResumeFailedException,
                        java.io.IOException

Throws:

StorageFormatException

ResumeFailedException

java.io.IOException

Method Detail

schedule

public void schedule(ClientContext context)

Description copied from interface: ClientGetState

Schedule the request on the ClientRequestScheduler.

Specified by:

schedule in interface ClientGetState

failOnDiskError

public void failOnDiskError(java.io.IOException e)

Fail the whole splitfile request when we get an IOException on writing to or reading from the on-disk storage. Can be called asynchronously by SplitFileFetcher*Storage if an off-thread job (e.g. FEC decoding) breaks, or may be called when SplitFileFetcher*Storage throws.

Specified by:

failOnDiskError in interface SplitFileFetcherStorageCallback

Parameters:

e - The IOException, generated when accessing the on-disk storage.

failOnDiskError

public void failOnDiskError(ChecksumFailedException e)

Fail the whole splitfile request when we get unrecoverable data corruption, e.g. can't read the keys. FIXME ROBUSTNESS in some cases this could actually be recovered by restarting from the metadata or the original URI.

Specified by:

failOnDiskError in interface SplitFileFetcherStorageCallback

fail

public void fail(FetchException e)

Description copied from interface: SplitFileFetcherStorageCallback

Called when the fetch failed, e.g. due to running out of retries.

Specified by:

fail in interface SplitFileFetcherStorageCallback

cancel

public void cancel(ClientContext context)

Description copied from interface: ClientGetState

Cancel the request, and call onFailure() on the callback in order to tell downstream (ultimately the client) that cancel has succeeded, and to allow it to call removeFrom() to avoid a database leak.

Specified by:

cancel in interface ClientGetState

getToken

public long getToken()

Description copied from interface: ClientGetState

Get a long value which may be passed around to identify this request (e.g. by the USK fetching code).

Specified by:

getToken in interface ClientGetState

onSuccess

public void onSuccess()

The splitfile download succeeded. Generate a stream and send it to the GetCompletionCallback. See bug #6063 for a better way that probably is too much complexity for the benefit.

Specified by:

onSuccess in interface SplitFileFetcherStorageCallback

onClosed

public void onClosed()

Description copied from interface: SplitFileFetcherStorageCallback

Called when the storage layer has finished, the higher level code has finished, and the storage has been freed, i.e. the request is now completely finished.

Specified by:

onClosed in interface SplitFileFetcherStorageCallback

getPriorityClass

public short getPriorityClass()

Description copied from interface: SplitFileFetcherStorageCallback

Get the priority class of the request. Needed for e.g. FEC decoding scheduling.

Specified by:

getPriorityClass in interface SplitFileFetcherStorageCallback

setSplitfileBlocks

public void setSplitfileBlocks(int requiredBlocks,
                               int remainingBlocks)

Description copied from interface: SplitFileFetcherStorageCallback

Called during construction to tell other layers how many blocks to expect.

Specified by:

setSplitfileBlocks in interface SplitFileFetcherStorageCallback

Parameters:

requiredBlocks - The number of blocks that must be fetched to complete the download.

remainingBlocks - The total number of blocks minus requiredBlocks.

onSplitfileCompatibilityMode

public void onSplitfileCompatibilityMode(InsertContext.CompatibilityMode min,
                                         InsertContext.CompatibilityMode max,
                                         byte[] customSplitfileKey,
                                         boolean compressed,
                                         boolean bottomLayer,
                                         boolean definitiveAnyway)

Description copied from interface: SplitFileFetcherStorageCallback

Called during construction, when we know the settings for the splitfile.

Specified by:

onSplitfileCompatibilityMode in interface SplitFileFetcherStorageCallback

Parameters:

min - The lowest CompatibilityMode that appears to be valid based on what we've fetched so far.

max - The highest CompatibilityMode that appears to be valid based on what we've fetched so far.

customSplitfileKey - The fixed byte[] encryption key used on insert. On anything recent, we generate a single key, randomly for an SSK, or based on the content for a CHK, and use it for everything. This saves metadata space and improves security for SSKs.

compressed - Whether the content is compressed. If false, the dontCompress option was used.

bottomLayer - Whether this report originates at the bottom layer of the splitfile pyramid. I.e. the actual file, not the file containing the metadata to fetch the file (this can recurse for several levels!)

definitiveAnyway - Whether this report is definitive even though it's not from the bottom layer. This is true of recent splitfiles, where we store all the data in the top key.

queueHeal

public void queueHeal(byte[] data,
                      byte[] cryptoKey,
                      byte cryptoAlgorithm)

Description copied from interface: SplitFileFetcherStorageCallback

Queue a block to be healed. LOCKING: Called on the decode thread, so should avoid taking any dangerous locks and not be too slow.

Specified by:

queueHeal in interface SplitFileFetcherStorageCallback

localRequestOnly

public boolean localRequestOnly()

toNetwork

public void toNetwork()

hasFinished

public boolean hasFinished()

onFetchedBlock

public void onFetchedBlock()

Specified by:

onFetchedBlock in interface SplitFileFetcherStorageCallback

onFailedBlock

public void onFailedBlock()

Description copied from interface: SplitFileFetcherStorageCallback

Called when the splitfile fetcher gives up on a block. (Assumed to be a non-fatal error, run out of retries)

Specified by:

onFailedBlock in interface SplitFileFetcherStorageCallback

onResume

public void onResume(int succeededBlocks,
                     int failedBlocks,
                     ClientMetadata meta,
                     long finalSize)

Specified by:

onResume in interface SplitFileFetcherStorageCallback

maybeAddToBinaryBlob

public void maybeAddToBinaryBlob(ClientCHKBlock block)

Description copied from interface: SplitFileFetcherStorageCallback

Called whenever we successfully download, decode or encode a block and it matches the expected key. LOCKING: Called on the decode thread so should avoid taking any dangerous locks.

Specified by:

maybeAddToBinaryBlob in interface SplitFileFetcherStorageCallback

wantBinaryBlob

public boolean wantBinaryBlob()

Description copied from interface: SplitFileFetcherStorageCallback

Do we want maybeAddToBinaryBlob() to be called?? LOCKING: Should not take any locks.

Specified by:

wantBinaryBlob in interface SplitFileFetcherStorageCallback

getSendableGet

public BaseSendableGet getSendableGet()

Description copied from interface: SplitFileFetcherStorageCallback

Can be null. Provided mainly for KeysFetchingLocally.

Specified by:

getSendableGet in interface SplitFileFetcherStorageCallback

restartedAfterDataCorruption

public void restartedAfterDataCorruption()

Description copied from interface: SplitFileFetcherStorageCallback

Called when we recover from disk corruption, and have to re-download some blocks that we had already downloaded but which were corrupted on disk. E.g. when a segment attempts to decode but discovers that a block doesn't match the key given.

Specified by:

restartedAfterDataCorruption in interface SplitFileFetcherStorageCallback

clearCooldown

public void clearCooldown()

Description copied from interface: SplitFileFetcherStorageCallback

Called when the fetcher may have exited cooldown early.

Specified by:

clearCooldown in interface SplitFileFetcherStorageCallback

reduceCooldown

public void reduceCooldown(long wakeupTime)

Description copied from interface: SplitFileFetcherStorageCallback

Called when the wakeup time reduces but it is still not fetchable.

Specified by:

reduceCooldown in interface SplitFileFetcherStorageCallback

getHasKeyListener

public HasKeyListener getHasKeyListener()

Description copied from interface: SplitFileFetcherStorageCallback

Can be null. Provided for KeyListeners.

Specified by:

getHasKeyListener in interface SplitFileFetcherStorageCallback

onResume

public void onResume(ClientContext context)
              throws FetchException

Description copied from interface: ClientGetState

Called on restarting the node for a persistent request. The request must re-schedule itself, as neither the KeyListener's nor the RGA's are persistent now.

Specified by:

onResume in interface ClientGetState

Throws:

FetchException

getSalter

public KeySalter getSalter()

Specified by:

getSalter in interface SplitFileFetcherStorageCallback

writeTrivialProgress

public boolean writeTrivialProgress(java.io.DataOutputStream dos)
                             throws java.io.IOException

Throws:

java.io.IOException

onShutdown

public void onShutdown(ClientContext context)

Description copied from interface: ClientGetState

Called just before the final write of client.dat before the node shuts down. Should write any dirty data to disk etc.

Specified by:

onShutdown in interface ClientGetState