The process of uploading files into the data cache can be automated. In order to write code to automate this you need two things: First you need to know what to put in the query strings used to access the cache - this document provides that information; Second you need a tool that allows you non-interactive access to the cache (e.g. from a script file). There are a number of such tools freely available - the examples in this document use the 'CURL' program, which is already installed on many UNIX systems, but is also available from http://curl.haxx.se/.

At the end of this document there are some examples of using CURL to access the cache. You can also download UNIX shell scripts that provide working example of using CURL to access the cache. In addition BGS has developed a Java program and library that provides an alternative way to access the cache.

URL and Query String Details

The three actions that can be performed on the cache are uploading data, searching the cache and downloading data. The third of these is only used by the GIN when ingesting data and is not relavant to observatories uploading data, but is documented here for completeness. All three actions occur through one URL. The absolute value of this URL depends on the base URL for the data cache application. As an example, if the application is installed at https://gin-upload.bgs.ac.uk/GINFileUpload, then access to the cache is through https://gin-upload.bgs.ac.uk/GINFileUpload/Cache. Parameters sepcifying the action you want to perform may be sent using http POST or http GET method, but note that file upload will only work with http POST.

The parameters are described here using the query string format associated with http GET. The application supports both POST and GET methods, except for file upload, where only POST is supported. The query string URL is relative to the absolute URL of the application. All query strings support the Format parameter, which may be set to either 'plain' or 'html' (the default is 'plain'). A format of 'plain' will result in plain text files (with no html formatting) being emitted.

The following format and type codes are supported in both the query string and the returned search results.

Responses to Cache Operations

Responses will be either in plain text or HTML (depending on the format set with the 'Format' field that is available for all cache operations). HTML responses are designed to be easy to read for an interactive user (and will not be described further). Plain text messages are designed to be simple to process from software.

The response to a file upload is one of two types. The first is sent when the data was loaded successfully:

    Success
    Data loaded OK, data file id = <ID>

The second is sent if there was an error loading the data:

    Error
    <Cache error message>

Data can also be bulk uploaded in a .zip or .gz file. In this case results are returned as two header lines plus a status line for each entry in the .zip or .gz file:

    <Status>
    Zip file, <N> entries
    1 (<Filename>) Success: Data loaded OK, data file id = <ID>
    2 (<Filename>) Error: <Cache error message>
    ... Up to the number of entries in the .zip or .gz file

The <Status> code for a bulk upload has the following meanings:

• Success: all data files in the archive were loaded successfully
• Warning: at least one data file was loaded successfully
• Error: no data files were successfully loaded

Plain text search results are formatted as follows. Each data file in the cache is represented by a row in the returned text file. Each row has the following fields:

    <ID> <Station code> <Type-code> <Data date/time> <Data duration> <Upload date/time>

All fields have a fixed width and are separated by a single space. Numbers less than the maximum width are padded to the left with spaces or zeroes. Alphanumeric strings less than the maximum width are padded to the right with spaces.

• The <ID> has a width of 19 characters.
• The <Station code> has a width of 3 characters.
• The <Type-Code> (described in the table above) has a field width of 1.
• The <Data duration>, which is specified in seconds, has a width of 5 (allowing for a maximum value of 86400 � or 24 hours).
• Date / time formats are as described in the 'search' query string.

The successful response to a download request is the data file.

Any of the cache services may break down, in which case an error response will be sent. The first line of an error response indicates the level of fault. It may contain one of the following words:

The second line of the response is a message describing the problem. The response may include subsequent lines with further detail.

    curl -F "File=@had20070531d.min;type=text/plain" -F "Format=plain" -F "Request=Upload" https://gin-upload.bgs.ac.uk/GINFileUpload/Cache

You will normally require authentication to use the cache. More information on the users who can access the cache and the roles they are allowed is given here. For example, if you are using the username 'imo' and the password 'dummy-password', add these options to the CURL command line:

    -u imo:dummy_password

The -u option can be added to all the CURL examples in this section, but has been left out to make the examples more readable.

Each item uploaded into the cache is given an ID. The IDs are consecutive, long (64-bit) numbers starting at 0. These IDs can be used to discover how many items have been uploaded since a previous call. To get a list of all files uploaded since (and including) ID 5500:

    curl -FRequest=Search -FSmallestID=5500 https://gin-upload.bgs.ac.uk/GINFileUpload/Cache

To find the earliest item in the cache:

    curl -FRequest=Search -FmaxNResults=1 https://gin-upload.bgs.ac.uk/GINFileUpload/Cache