freenet.client.async

Class SplitFileInserterStorage

java.lang.Object

freenet.client.async.SplitFileInserterStorage

public class SplitFileInserterStorage
extends java.lang.Object

Similar to SplitFileFetcherStorage. The status of a splitfile insert, including the encoded check and cross-check blocks, is stored in a RandomAccessBuffer separate to that containing the data itself. This class is only concerned with encoding, storage etc, and should have as few dependencies on the runtime as possible, partly so that it can be tested in isolation. Actual details of inserting the blocks will be kept on SplitFileInserter. CONTENTS OF FILE: (for persistent case, transient leaves out most of the below) * Magic, version etc. * Basic settings. * Settings for each segment. * Settings for each cross-segment (including block lists). * Padded last block (if necessary) * Global status * (Padding to a 4096 byte boundary) * Cross-check blocks (by cross-segment). * Check blocks * Segment status * Segment key list (each key separately stored and checksummed) Intentionally not as robust (against disk corruption in particular) as SplitFileFetcherStorage: We encode the block CHKs when we encode the check blocks, and if the block data has changed by the time we insert the data, we fail; there's no way to obtain correct data. Similarly, we fail if the keys are corrupted. However, for most of the duration of the insert neither the data, the check blocks or the keys will be written, so it's still reasonable assuming you only get errrors when writing, and it isn't really possible to improve on this much anyway...

Constructor Summary

Constructors

Constructor and Description
SplitFileInserterStorage(LockableRandomAccessBuffer raf, LockableRandomAccessBuffer originalData, SplitFileInserterStorageCallback callback, java.util.Random random, MemoryLimitedJobRunner memoryLimitedJobRunner, PersistentJobRunner jobRunner, Ticker ticker, KeysFetchingLocally keysFetching, FilenameGenerator persistentFG, PersistentFileTracker persistentFileTracker, MasterSecret masterKey) Create a splitfile insert from stored data.
SplitFileInserterStorage(LockableRandomAccessBuffer originalData, long decompressedLength, SplitFileInserterStorageCallback callback, Compressor.COMPRESSOR_TYPE compressionCodec, ClientMetadata meta, boolean isMetadata, ArchiveManager.ARCHIVE_TYPE archiveType, LockableRandomAccessBufferFactory rafFactory, boolean persistent, InsertContext ctx, byte splitfileCryptoAlgorithm, byte[] splitfileCryptoKey, byte[] hashThisLayerOnly, HashResult[] hashes, BucketFactory bf, ChecksumChecker checker, java.util.Random random, MemoryLimitedJobRunner memoryLimitedJobRunner, PersistentJobRunner jobRunner, Ticker ticker, KeysFetchingLocally keysFetching, boolean topDontCompress, int topRequiredBlocks, int topTotalBlocks, long origDataSize, long origCompressedDataSize) Create a SplitFileInserterStorage.

Method Summary

Modifier and Type	Method and Description
void	addFailure(InsertException e)
freenet.client.async.SplitFileInserterSegmentStorage.BlockInsert	chooseBlock() Choose a block to insert.
void	clearCooldown()
long	countAllKeys()
int	countEncodedCrossSegments()
int	countEncodedSegments()
long	countSendableKeys()
void	failFatalErrorInBlock()
void	failOnDiskError(java.io.IOException e)
void	failTooManyRetriesInBlock()
long	getOffsetSegmentStatus(int segNo)
freenet.client.async.SplitFileInserterStorage.Status	getStatus()
int	getTotalBlockCount()
long	getWakeupTime(ClientContext context, long now)
boolean	hasFinished()
boolean	hasSplitfileKey()
protected boolean	isFinishing()
void	lazyWriteMetadata()
boolean	noBlocksToSend()
void	onFinishedEncoding(SplitFileInserterCrossSegmentStorage completed) Called when a cross-segment finishes encoding blocks.
void	onFinishedEncoding(SplitFileInserterSegmentStorage completed) Called when a segment finishes encoding blocks.
void	onHasKeys(SplitFileInserterSegmentStorage splitFileInserterSegmentStorage)
void	onResume(ClientContext context)
byte[]	readCheckBlock(int segNo, int checkBlockNo)
byte[]	readSegmentCheckBlock(int segNo, int checkBlockNo)
byte[]	readSegmentDataBlock(int segNo, int blockNo)
void	segmentSucceeded(SplitFileInserterSegmentStorage completedSegment) Called when a segment completes.
void	start()
int	totalCrossCheckBlocks()
void	writeSegmentCheckBlock(int segNo, int checkBlockNo, byte[] buf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SplitFileInserterStorage

public SplitFileInserterStorage(LockableRandomAccessBuffer originalData,
                                long decompressedLength,
                                SplitFileInserterStorageCallback callback,
                                Compressor.COMPRESSOR_TYPE compressionCodec,
                                ClientMetadata meta,
                                boolean isMetadata,
                                ArchiveManager.ARCHIVE_TYPE archiveType,
                                LockableRandomAccessBufferFactory rafFactory,
                                boolean persistent,
                                InsertContext ctx,
                                byte splitfileCryptoAlgorithm,
                                byte[] splitfileCryptoKey,
                                byte[] hashThisLayerOnly,
                                HashResult[] hashes,
                                BucketFactory bf,
                                ChecksumChecker checker,
                                java.util.Random random,
                                MemoryLimitedJobRunner memoryLimitedJobRunner,
                                PersistentJobRunner jobRunner,
                                Ticker ticker,
                                KeysFetchingLocally keysFetching,
                                boolean topDontCompress,
                                int topRequiredBlocks,
                                int topTotalBlocks,
                                long origDataSize,
                                long origCompressedDataSize)
                         throws java.io.IOException,
                                InsertException

Create a SplitFileInserterStorage.

Parameters:

originalData - The original data as a RandomAccessBuffer. We need to be able to read single blocks.

rafFactory - A factory (persistent or not as appropriate) used to create the temporary RAF storing check blocks etc.

bf - A temporary Bucket factory used for temporarily storing the cross-segment settings.

topRequiredBlocks - The minimum number of blocks needed to fetch the content, so far. We will add to this for the final metadata.

Throws:

java.io.IOException

InsertException

SplitFileInserterStorage

public SplitFileInserterStorage(LockableRandomAccessBuffer raf,
                                LockableRandomAccessBuffer originalData,
                                SplitFileInserterStorageCallback callback,
                                java.util.Random random,
                                MemoryLimitedJobRunner memoryLimitedJobRunner,
                                PersistentJobRunner jobRunner,
                                Ticker ticker,
                                KeysFetchingLocally keysFetching,
                                FilenameGenerator persistentFG,
                                PersistentFileTracker persistentFileTracker,
                                MasterSecret masterKey)
                         throws java.io.IOException,
                                StorageFormatException,
                                ChecksumFailedException,
                                ResumeFailedException

Create a splitfile insert from stored data.

Parameters:

raf - The file the insert was stored to. Caller must resume it before calling constructor.

originalData - The original data to be inserted. Caller must resume it before calling constructor.

callback - The parent callback (e.g. SplitFileInserter).

random -

memoryLimitedJobRunner -

jobRunner -

ticker -

keysFetching -

persistentFG -

persistentFileTracker -

masterKey -

Throws:

java.io.IOException

StorageFormatException

ChecksumFailedException

ResumeFailedException

Method Detail

start

public void start()

countEncodedSegments

public int countEncodedSegments()

countEncodedCrossSegments

public int countEncodedCrossSegments()

onFinishedEncoding

public void onFinishedEncoding(SplitFileInserterCrossSegmentStorage completed)

Called when a cross-segment finishes encoding blocks. Can be called inside locks as it runs off-thread.

Parameters:

completed -

onFinishedEncoding

public void onFinishedEncoding(SplitFileInserterSegmentStorage completed)

Called when a segment finishes encoding blocks. Can be called inside locks as it runs off-thread.

Parameters:

completed -

onHasKeys

public void onHasKeys(SplitFileInserterSegmentStorage splitFileInserterSegmentStorage)

hasSplitfileKey

public boolean hasSplitfileKey()

readCheckBlock

public byte[] readCheckBlock(int segNo,
                             int checkBlockNo)
                      throws java.io.IOException

Throws:

java.io.IOException

readSegmentDataBlock

public byte[] readSegmentDataBlock(int segNo,
                                   int blockNo)
                            throws java.io.IOException

Throws:

java.io.IOException

writeSegmentCheckBlock

public void writeSegmentCheckBlock(int segNo,
                                   int checkBlockNo,
                                   byte[] buf)
                            throws java.io.IOException

Throws:

java.io.IOException

readSegmentCheckBlock

public byte[] readSegmentCheckBlock(int segNo,
                                    int checkBlockNo)
                             throws java.io.IOException

Throws:

java.io.IOException

totalCrossCheckBlocks

public int totalCrossCheckBlocks()

segmentSucceeded

public void segmentSucceeded(SplitFileInserterSegmentStorage completedSegment)

Called when a segment completes. Can be called inside locks as it runs off-thread.

addFailure

public void addFailure(InsertException e)

failOnDiskError

public void failOnDiskError(java.io.IOException e)

failFatalErrorInBlock

public void failFatalErrorInBlock()

failTooManyRetriesInBlock

public void failTooManyRetriesInBlock()

hasFinished

public boolean hasFinished()

getStatus

public freenet.client.async.SplitFileInserterStorage.Status getStatus()

lazyWriteMetadata

public void lazyWriteMetadata()

isFinishing

protected boolean isFinishing()

getOffsetSegmentStatus

public long getOffsetSegmentStatus(int segNo)

onResume

public void onResume(ClientContext context)
              throws ResumeFailedException

Throws:

ResumeFailedException

chooseBlock

public freenet.client.async.SplitFileInserterSegmentStorage.BlockInsert chooseBlock()

Choose a block to insert. FIXME make SplitFileInserterSender per-segment, eliminate a lot of unnecessary complexity.

noBlocksToSend

public boolean noBlocksToSend()

countAllKeys

public long countAllKeys()

countSendableKeys

public long countSendableKeys()

getTotalBlockCount

public int getTotalBlockCount()

clearCooldown

public void clearCooldown()

getWakeupTime

public long getWakeupTime(ClientContext context,
                          long now)

Returns:

-1 if the insert has finished, 0 if has blocks to send, otherwise Long.MAX_VALUE.