katsdpmodels.fetch package

Submodules

katsdpmodels.fetch.aiohttp module

katsdpmodels.fetch.requests module

Fetch models over HTTP.

class katsdpmodels.fetch.requests.Fetcher(*, session: Optional[requests.sessions.Session] = None, model_cache: Optional[MutableMapping[str, katsdpmodels.models.Model]] = None)

Bases: katsdpmodels.fetch.FetcherBase

Fetches and caches models, using the requests library.

It caches every URL it fetches (ignoring any cache control headers), so it should not be reused over a long time. It is best suited to fetching a batch of models, some of which may turn out to be aliases of each other.

It should be closed with close() when no longer in use. It also implements the context manager protocol for this purpose. This will close the retrieved models, so must only be done once the models are no longer in use.

This class is not thread-safe.

Parameters

  • session – Interface for doing the actual HTTP requests. If not specified, an internal session will be created (which supports file:// URLs), and closed when the fetcher is closed. If a custom session is provided it will not be closed (so it can be shared between multiple Fetcher instances).

  • model_cache – A dictionary for caching models by URL. This is not typically needed, as the fetcher will use an internal cache if one is not provided, but allows fetchers to share a cache (but not in a thread-safe way!). If a custom cache is provided, then close() will not close the models in it, and the caller is responsible for doing so.

Raises
close() → None
get(url: str, model_class: Type[_M], *, lazy: bool = False) → _M

Retrieve a single model.

The caller must not close the retrieved model, as it is cached and a future request for the model would return the closed model. Instead, the caller must close the fetcher once it no longer needs any of the models.

Parameters

  • url – The URL to fetch.

  • model_class – The base class for the model type (do not use concrete format-specific classes). The returned model will be of this type.

  • lazy –

    If true, create a view of the HDF5 file that only retrieves data as it is needed. Whether this actually allows data to be loaded lazily depends on the model_class: some model classes will read all the data into memory on creation, in which case lazy loading may perform significantly worse (at the time of writing, no model classes support lazy loading).

    Lazy loading imposes some additional requirements:

    1. The session must not be closed while the model is in use, even if the fetcher is no longer needed.

    2. The checksum stored in the filename is not validated.

    3. If the model is already in the cache, the laziness setting is ignored and the cached model is returned.

Raises

  • ModelError – If there are high-level errors with the model.

  • requests.exception.RequestException – Any exceptions raised by the underlying session.

resolve(url: str) → List[str]

Follow a chain of aliases.

Return a list of URLs found along the chain. The first element is the given URL and the final element is the resolved model.

Raises

TooManyAliasesError – If there were more than MAX_ALIASES aliases or a cycle was found.

property session
class katsdpmodels.fetch.requests.HttpFile(session: requests.sessions.Session, url: str)

Bases: io.RawIOBase

File-like object that fetches byte ranges via HTTP.

This requires the server to advertise support for byte-range requests and to provide a Content-Length. It is currently not robust against the content changing.

Raises
close() → None

Flush and close the IO object.

This method has no effect if the file is already closed.

readinto(b) → int
seek(offset: int, whence: int = 0) → int

Change stream position.

Change the stream position to the given byte offset. The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • 0 – start of stream (the default); offset should be zero or positive

  • 1 – current stream position; offset may be negative

  • 2 – end of stream; offset is usually negative

Return the new absolute position.

tell() → int

Return current stream position.

property url

Return the final URL.

This may differ from the constructor argument if HTTP redirects occurred.

class katsdpmodels.fetch.requests.TelescopeStateFetcher(telstate: katsdptelstate.TelescopeState, fetcher: Optional[katsdpmodels.fetch.requests.Fetcher] = None)

Bases: katsdpmodels.fetch.TelescopeStateFetcherBase[katsdptelstate.TelescopeState]

Fetches models that are referenced by telescope state.

The telescope state must have a sdp_model_base_url key with a base URL, and a key per model with an URL relative to this one. If it is missing then a KeyError will be raised from get(), rather than the constructor.

If no fetcher is provided, an internal one will be created, and closed when this object is closed.

This class is not thread-safe.

close() → None

Clean up resources.

get(key: str, model_class: Type[_M], *, telstate: Optional[katsdptelstate.TelescopeState] = None, lazy: bool = False) → _M

Retrieve a single model.

The semantics are the same as for Fetcher.get(). Any problems with getting the keys from the telescope state will raise TelescopeStateError.

If telstate is provided, it is used instead of the constructor argument for fetching the model key; but the constructor telstate is still used to fetch sdp_model_base_url.

katsdpmodels.fetch.requests.fetch_model(url: str, model_class: Type[_M], *, session: Optional[requests.sessions.Session] = None) → _M

Retrieve a single model.

This is a convenience function that should only be used when loading just a single model. If multiple models will be used instead, construct an instance of Fetcher and use it to fetch models, as this will allow models that turn out to be the same to be shared.

Module contents

class katsdpmodels.fetch.FetcherBase(*, model_cache: Optional[MutableMapping[str, katsdpmodels.models.Model]] = None)

Bases: object

Base class for HTTP fetcher implementations.

It caches every URL it fetches (ignoring any cache control headers), so it should not be reused over a long time. It is best suited to fetching a batch of models, some of which may turn out to be aliases of each other.

It does not perform any I/O itself. Instead, it provides generators that yield Requests and expects to receive Responses in reply. The subclass is responsible for producing the responses to requests. This design allows the core logic to be shared between synchronous and asynchronous implementations.

Parameters

model_cache – A dictionary for caching models by URL. This is not typically needed, as the fetcher will use an internal cache if one is not provided, but allows fetchers to share a cache (but not in a thread-safe way!). If a custom cache is provided, then closing the fetcher will not close the models in it, and the caller is responsible for doing so.

class katsdpmodels.fetch.FileResponse(url: str, headers: Mapping[str, str], file: Union[io.IOBase, BinaryIO], content: Optional[bytes] = None)

Bases: katsdpmodels.fetch.Response

Response to a ResponseType.FILE request.

Parameters

  • url – The final URL of the request, after handling any redirects.

  • headers – A case-insensitive mapping of HTTP headers

  • file – A file-like object from which the binary content can be read. The receiver of the response is responsible for closing it.

  • content – If available, the content of the response. It is used only for verifying checksums.

katsdpmodels.fetch.MAX_ALIASES = 30

Maximum number of aliases that will be followed to find a model

class katsdpmodels.fetch.Request(url: str, response_type: katsdpmodels.fetch.ResponseType)

Bases: object

Bare essentials of an HTTP request.

class katsdpmodels.fetch.Response(url: str, headers: Mapping[str, str])

Bases: object

Bare essentials of a response to a Request.

Parameters

  • url – The final URL of the request, after handling any redirects.

  • headers – A case-insensitive mapping of HTTP headers

property content_type

Return the MIME content type of the request.

Any parameters (like encoding) are stripped off. If the Content-Type header is missing or it is application/octet-stream, returns None.

class katsdpmodels.fetch.ResponseType(value)

Bases: enum.Enum

An enumeration.

FILE = 1
TEXT = 0
class katsdpmodels.fetch.TelescopeStateFetcherBase(telstate: _TS)

Bases: Generic[typing._TS]

Fetches models that are referenced by telescope state.

The telescope state must have a sdp_model_base_url key with a base URL, and a key per model with an URL relative to this one. If it is missing then a KeyError will be raised from get(), rather than the constructor.

If no fetcher is provided, an internal one will be created, and closed when this object is closed.

This class is not thread-safe.

class katsdpmodels.fetch.TelescopeStateRequest(telstate: _TS, key: str)

Bases: Generic[typing._TS]

A request for a key from a telescope state.

class katsdpmodels.fetch.TextResponse(url: str, headers: Mapping[str, str], text: str)

Bases: katsdpmodels.fetch.Response

Response to a ResponseType.TEXT request.