freenet.client.async

Class SplitFileFetcherSegmentStorage

java.lang.Object

freenet.client.async.SplitFileFetcherSegmentStorage

public class SplitFileFetcherSegmentStorage
extends java.lang.Object

Represents a single segment, in memory and on disk. Handles storage and decoding. Note that the on-disk data, and therefore the read-in metadata, may be inaccurate; we check everything opportunistically. Hence we are very robust (but not completely immune) to disk corruption.

See Also:

SplitFileFetcherStorage

Field Summary

Fields

Modifier and Type	Field and Description
int	checkBlocks Count of check blocks (generated by FEC).
int	crossSegmentCheckBlocks Count of cross-segment check blocks.
int	dataBlocks Count of data blocks (actual data divided up into CHKs, though the last one will be padded).

Constructor Summary

Constructors

Constructor and Description
SplitFileFetcherSegmentStorage(SplitFileFetcherStorage parent, java.io.DataInputStream dis, int segNo, boolean writeRetries, long segmentDataOffset, long segmentCrossCheckDataOffset, long segmentKeysOffset, long segmentStatusOffset, KeysFetchingLocally keysFetching) Construct from a saved file.
SplitFileFetcherSegmentStorage(SplitFileFetcherStorage parent, int segNumber, Metadata.SplitfileAlgorithm splitfileType, int dataBlocks, int checkBlocks, int crossCheckBlocks, long segmentDataOffset, long segmentCrossCheckDataOffset, long segmentKeysOffset, long segmentStatusOffset, boolean writeRetries, SplitFileSegmentKeys keys, KeysFetchingLocally keysFetching) Construct a segment.

Method Summary

Modifier and Type	Method and Description
int	allocateCrossCheckBlock(SplitFileFetcherCrossSegmentStorage seg, java.util.Random random) Allocate a cross-segment check block.
int	allocateCrossDataBlock(SplitFileFetcherCrossSegmentStorage seg, java.util.Random random) Allocate a cross-segment data block.
void	cancel()
byte[]	checkAndGetBlockData(int blockNum)
int	chooseRandomKey() Pick a key to fetch.
long	countSendableKeys(long now, int maxRetries)
long	countUnfetchedKeys()
boolean	definitelyWantKey(NodeCHK key)
int	failedBlocks()
int	foundBlocks()
ClientCHK	getKey(int blockNum)
long	getOverallCooldownTime()
SplitFileSegmentKeys	getSegmentKeys()
void	getUnfetchedKeys(java.util.List<Key> keys)
boolean	hasBlock(int blockNo)
boolean	hasSucceeded()
boolean	isDecoding()
boolean	isDecodingOrFinished()
boolean	isFinished()
boolean	needsDecode()
void	onFinishedCheckingDatastoreNoFetch(ClientContext context) Called after checking datastore for a datastore-only request.
boolean	onGotKey(NodeCHK key, CHKBlock block) A block has been fetched which the caller believes is one of ours.
void	onNonFatalFailure(int blockNumber)
static int	paddedStoredSegmentStatusLength(int dataBlocks, int checkBlocks, int crossCheckBlocks, boolean trackRetries, int checksumLength, boolean persistent)
static int	storedSegmentStatusLength(int dataBlocks, int checkBlocks, int crossCheckBlocks, boolean trackRetries)
boolean	tryStartDecode() Read all the blocks, encode them according to their supposed keys and check that they are in fact the blocks that they should be.
void	writeFixedMetadata(java.io.DataOutputStream dos) Write minimal fixed metadata for the segment.
void	writeMetadata() Write the status metadata to disk, after a series of updates.
void	writeMetadata(boolean force) Write the status metadata to disk, after a series of updates.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

dataBlocks

public final int dataBlocks

Count of data blocks (actual data divided up into CHKs, though the last one will be padded). Numbered 0 .. dataBlocks-1.

crossSegmentCheckBlocks

public final int crossSegmentCheckBlocks

Count of cross-segment check blocks. These occur only in larger splitfiles and count as data blocks for segment-level FEC, but they also count as check blocks for cross-segment level FEC. Generally between 0 and 3. Numbered dataBlocks .. dataBlocks+crossSegmentCheckBlocks-1

checkBlocks

public final int checkBlocks

Count of check blocks (generated by FEC). Numbered dataBlocks+crossSegmentCheckBlocks .. dataBlocks+crossSegmentCheckBlocks+checkBlocks-1

Constructor Detail

SplitFileFetcherSegmentStorage

public SplitFileFetcherSegmentStorage(SplitFileFetcherStorage parent,
                                      int segNumber,
                                      Metadata.SplitfileAlgorithm splitfileType,
                                      int dataBlocks,
                                      int checkBlocks,
                                      int crossCheckBlocks,
                                      long segmentDataOffset,
                                      long segmentCrossCheckDataOffset,
                                      long segmentKeysOffset,
                                      long segmentStatusOffset,
                                      boolean writeRetries,
                                      SplitFileSegmentKeys keys,
                                      KeysFetchingLocally keysFetching)

Construct a segment.

Parameters:

parent -

segNumber -

splitfileType -

dataBlocks -

checkBlocks -

crossCheckBlocks -

segmentDataOffset -

segmentCrossCheckDataOffset - -1 to mean store the cross-check blocks just after the data and check blocks for this segment. Otherwise the offset.

segmentKeysOffset -

segmentStatusOffset -

writeRetries -

keys -

SplitFileFetcherSegmentStorage

public SplitFileFetcherSegmentStorage(SplitFileFetcherStorage parent,
                                      java.io.DataInputStream dis,
                                      int segNo,
                                      boolean writeRetries,
                                      long segmentDataOffset,
                                      long segmentCrossCheckDataOffset,
                                      long segmentKeysOffset,
                                      long segmentStatusOffset,
                                      KeysFetchingLocally keysFetching)
                               throws java.io.IOException,
                                      StorageFormatException

Construct from a saved file. Uses the DataInputStream to read static settings, i.e. number of blocks, does not use the RAF to read block status etc; caller must call readMetadata and readKeys separately for that.

Parameters:

parent -

dis - DataInputStream to which the static settings have been saved. Anything else we will need to read separately from the RandomAccessBuffer.

segNo - The segment number.

writeRetries -

segmentDataOffset -

segmentCrossCheckDataOffset - -1 to mean store the cross-check blocks just after the data and check blocks for this segment. Otherwise the offset.

segmentKeysOffset -

segmentStatusOffset -

keysFetching -

Throws:

java.io.IOException

StorageFormatException

Method Detail

getSegmentKeys

public SplitFileSegmentKeys getSegmentKeys()
                                    throws java.io.IOException

Throws:

java.io.IOException

writeMetadata

public void writeMetadata()
                   throws java.io.IOException

Write the status metadata to disk, after a series of updates.

Throws:

java.io.IOException

writeMetadata

public void writeMetadata(boolean force)
                   throws java.io.IOException

Write the status metadata to disk, after a series of updates.

Throws:

java.io.IOException

tryStartDecode

public boolean tryStartDecode()

Read all the blocks, encode them according to their supposed keys and check that they are in fact the blocks that they should be. If the metadata is inaccurate, update it and writeMetadata(). If we have enough blocks to decode, and we don't have all the blocks, then schedule a decode on the FEC thread.

Returns:

True if we scheduled a decode or are already finished. False if we do not have enough blocks to decode and need to fetch more blocks.

onGotKey

public boolean onGotKey(NodeCHK key,
                        CHKBlock block)
                 throws java.io.IOException

A block has been fetched which the caller believes is one of ours. Check whether it is in fact ours, and that we don't have it already. Find the key and decode it, and add it to our collection. If any cross-segments are waiting for this block, tell them. If we can decode, do so. Can be quite involved, should be called off-thread.

Parameters:

key -

block -

Returns:

True if we successfully decoded a block, in which case the function will be called again. False if there was no match, if we have already fetched that block, or if various errors occurred.

Throws:

java.io.IOException - If we were unable to write the block to disk.

storedSegmentStatusLength

public static int storedSegmentStatusLength(int dataBlocks,
                                            int checkBlocks,
                                            int crossCheckBlocks,
                                            boolean trackRetries)

paddedStoredSegmentStatusLength

public static int paddedStoredSegmentStatusLength(int dataBlocks,
                                                  int checkBlocks,
                                                  int crossCheckBlocks,
                                                  boolean trackRetries,
                                                  int checksumLength,
                                                  boolean persistent)

isFinished

public boolean isFinished()

isDecodingOrFinished

public boolean isDecodingOrFinished()

hasSucceeded

public boolean hasSucceeded()

onNonFatalFailure

public void onNonFatalFailure(int blockNumber)

allocateCrossDataBlock

public int allocateCrossDataBlock(SplitFileFetcherCrossSegmentStorage seg,
                                  java.util.Random random)

Allocate a cross-segment data block. Note that this algorithm must be reproduced exactly for splitfile compatibility; the Random seed is actually determined by the splitfile metadata.

Parameters:

seg - The cross-segment to allocate a block for.

random - PRNG seeded from the splitfile metadata, which determines which blocks to allocate in a deterministic manner.

Returns:

The data block number allocated.

allocateCrossCheckBlock

public int allocateCrossCheckBlock(SplitFileFetcherCrossSegmentStorage seg,
                                   java.util.Random random)

Allocate a cross-segment check block. Note that this algorithm must be reproduced exactly for splitfile compatibility; the Random seed is actually determined by the splitfile metadata.

Parameters:

seg - The cross-segment to allocate a block for.

random - PRNG seeded from the splitfile metadata, which determines which blocks to allocate in a deterministic manner.

Returns:

The block number allocated (between dataBlocks and dataBlocks+crossSegmentCheckBlocks).

definitelyWantKey

public boolean definitelyWantKey(NodeCHK key)

writeFixedMetadata

public void writeFixedMetadata(java.io.DataOutputStream dos)
                        throws java.io.IOException

Write minimal fixed metadata for the segment. This should include lengths rather than offsets. Does not write cross-segment block assignments; these are handled by the cross-segments.

Throws:

java.io.IOException

countUnfetchedKeys

public long countUnfetchedKeys()

countSendableKeys

public long countSendableKeys(long now,
                              int maxRetries)

getUnfetchedKeys

public void getUnfetchedKeys(java.util.List<Key> keys)
                      throws java.io.IOException

Throws:

java.io.IOException

chooseRandomKey

public int chooseRandomKey()

Pick a key to fetch. Must not update any persistent field. (Cooldowns etc are fine)

cancel

public void cancel()

getOverallCooldownTime

public long getOverallCooldownTime()

needsDecode

public boolean needsDecode()

foundBlocks

public int foundBlocks()

failedBlocks

public int failedBlocks()

getKey

public ClientCHK getKey(int blockNum)

checkAndGetBlockData

public byte[] checkAndGetBlockData(int blockNum)
                            throws java.io.IOException

Throws:

java.io.IOException

hasBlock

public boolean hasBlock(int blockNo)

isDecoding

public boolean isDecoding()

onFinishedCheckingDatastoreNoFetch

public void onFinishedCheckingDatastoreNoFetch(ClientContext context)

Called after checking datastore for a datastore-only request.