Implementation overview

Code organization

Each instrument is defined within its own module, a submodule of InstrumentControl.

Each instrument is a subtype of Instrument. By convention, instrument model numbers are used for module definitions (e.g. AWG5014C), so type names have "Ins" prepended (e.g. InsAWG5014).

Low-level wrappers for shared libraries are kept in their own packages (e.g. VISA, Alazar, and KeysightInstruments calls).

This way, at least some code can be reused if someone else does not want to use our codebase.

All sweep related type definitions and functions described in Sweep Jobs

can be found in src/Sweep.jl

Abstract type definitions like Instrument and Stimulus, are defined in

ICCommon.jl

src/Definitions.jl contains some definitions of other commonly used functions

and types. src/config.jl parses information in deps/config.json for talking to the database set up by ICDataServer, such as username information, database server address, path for saving results of sweeps, and stores it in a dictionary for access to all other functions that need this information to communicate with the database

Communication with ICDataServer

The functionality of the InstrumentControl.jl package is intertwined with the ICDataServer.jl package. ICDataServer sets up a relational database (RDBMS) with which it communicates with through SQL. InstrumentControl talks to that database; this database is used to maintain a log of information for each job: the job is identified by the job ID, and it mantains any metadata specified by the database creator. In its current implementation, the data saved to the database is the time of job submission, the time of job completion, and the latest job status, but we hope to add more logging functionality to the code over time.

When sweep is executed, the function communicates with the ICDataServer to create a new entry in the database table; the identifier of the new entry is a new job ID that the RDMS itself creates. ICDataServer then communicates back to InstrumentControl with this particular job ID and the time of submission; this job metadata is immediately stored in the SweepJob object (created by the sweep function as a handle to the new job). The job is then queued with the provided job ID as it's identifier.

The actual communication between the two packages is mediated by the popular ZeroMQ distributed messaging software; we utilize it's ZMQ Julia interface. While the reader is encouraged to go to these links for in-depth information, what you essentially need to communicate between a client and a server with ZeroMQ is a Context and a socket. In Julia, a ZMQ.Context object provides the framework for communication via a TCP connection (or any analagous form of communication). The client and server respectively will connect to a TCP port to send/receive information. The point of entry/exit for information being passed along this TCP connection are the ZMQ.Socket objects; they "bind" to the TCP ports and are the objects that the user actually calls on the send and receive information.

When InstrumentControl is imported, a ZMQ.Context object is automatically initialized. ZMQ.Socket objects are initialized in the first instance of communication with the ICDataServer, and the same object is used thereafter for communication within the same usage session. The socket objects are automatically bound to TCP ports that the user specifies in the deps/config.json file. InstrumentControl and ICDataServer communicate by binding to the same TCP connection.

Metaprogramming for VISA instruments

Many commercial instruments support a common communications protocol and command syntax (VISA and SCPI respectively). For such instruments, methods for setindex! and getindex, as well as Instrument subtype and InstrumentProperty subtype definitions, can be generated with metaprogramming, rather than typing them out explicitly.

The file src/MetaprogrammingVISA.jl is used heavily for code generation based on JSON template files. Since much of the logic for talking to instruments is the same between VISA instruments, in some cases no code needs to be written to control a new instrument provided an appropriate template file is prepared. The metaprogramming functions are described below although they are not intended to be used interactively.

# InstrumentControl.insjson — Function.

insjson(file::AbstractString)

Parses a JSON file with a standardized schema to describe how to control an instrument.

Here is an example of a valid JSON file with valid schema for parsing:

{
    "instrument":{
            "module":"E5071C",
            "type":"InsE5071C",
            "make":"Keysight",
            "model":"E5071C",
            "writeterminator":"\n"
    },
    "properties":[
        {
            "cmd":":CALCch:TRACtr:CORR:EDEL:TIME",
            "type":"VNA.ElectricalDelay",
            "values":[
                "v::Real"
            ],
            "infixes":[
                "ch::Integer=1",
                "tr::Integer=1"
            ],
            "doc": "My documentation"
        }
    ]
}

JSON.parse takes such a file and makes an 'instrument' dictionary and a 'properties' array. The instrument dictionary is described in the @generate_instruments documentation. The properties array contains one or more dictionaries, each with keys:

cmd: Specifies what must be sent to the instrument (it should be terminated

with "?" for query-only commands). The lower-case characters are replaced by "infixes", which are either numerical arguments or strings

type: Specifies the InstrumentProperty subtype to use cmd.
values: Specifies the required argument for setindex!, which will appear

after cmd in the string sent to the instrument.

infixes: Specifies the infix arguments to be put in cmd. This key is not

required if there are no infixes.

doc: Specifies documentation for the generated Julia functions. This key

is not required if there is no documentation. This is used not only for interactive help but also in generating the documentation you are reading.

The value of the properties.type field and entries in the properties.values and properties.infixes arrays are parsed into expressions or symbols for further manipulation. All generated dictionary keys are also converted to symbols for further manipulation.

source

# InstrumentControl.@generate_instruments — Macro.

@generate_instruments(metadata)

This macro takes a dictionary of metadata, typically obtained from a call to insjson. It operates on the :instrument field of the dictionary which is expected to have the following structure:

module: The module name. Can already exist but is created if it does not.

This field is converted from a string to a Symbol by insjson.

type: The name of the type to create for the new instrument.

This field is converted from a string to a Symbol by insjson.

super: This field is optional. If provided it will be the supertype of

the new instrument type, otherwise the supertype will be Instrument. This field is converted from a string to a Symbol by insjson.

make: The make of the instrument, e.g. Keysight, Tektronix, etc.
model: The model of the instrument, e.g. E5071C, AWG5014C, etc.
writeterminator: Write termination string for sending SCPI commands.

The macro imports required modules and methods, defines and exports the Instrument subtype, and defines and exports and the make and model methods if they do not exist already (note generic functions make and model are defined in src/Definitions.jl).

By convention we typically have the module name be the same as the model name, and the type is just the model prefixed by "Ins", e.g. InsE5071C. This is not required.

source

# InstrumentControl.@generate_properties — Macro.

@generate_properties(metadata)

This macro takes a dictionary of metadata, typically obtained from a call to insjson. It operates on the :properties field of the dictionary, which is expected to be a list of dictionaries with information on each "property" of the instrument. This macro specifically operates on the :type field of each property dictionary; this field contains the name of the type we would like to assign to a given property of the instrument.

For every property dictionary, the macro first checks if a type with name corresponding to the dictionary's :type field has already been defined. If not, it then defines an abstract type with that name, and makes it a subtype of the InstrumentProperty type defined in the ICCommon package. The macro then finally exports that type

source

# InstrumentControl.@generate_handlers — Macro.

@generate_handlers(instype, p)

This macro takes a symbol instype bound to an Instrument subtype (i.e. if the symbol was evaluated, it would return an Instrument subtype ), and a property dictionary p located in the :properties field of the dictionary of metadata generated by a call to insjson. with the auxiliary JSON file described above.

This macro is written to handle the cases where an instrument command does not accept numerical arguments, but rather a small set of options. Here is an example of the property dictionary (prior to parsing) for such a command, which sets/gets the format for a given channel and trace on the E5071C vector network analyzer:

{
    "cmd":":CALCch:TRACtr:FORM",
    "type":"VNAFormat",
    "values":[
        "v::Symbol in symbols"
    ],
    "symbols":{
        "LogMagnitude":"MLOG",
        "Phase":"PHAS",
        "GroupDelay":"GDEL",
        "SmithLinear":"SLIN",
        "SmithLog":"SLOG",
        "SmithComplex":"SCOM",
        "Smith":"SMIT",
        "SmithAdmittance":"SADM",
        "PolarLinear":"PLIN",
        "PolarLog":"PLOG",
        "PolarComplex":"POL",
        "LinearMagnitude":"MLIN",
        "SWR":"SWR",
        "RealPart":"REAL",
        "ImagPart":"IMAG",
        "ExpandedPhase":"UPH",
        "PositivePhase":"PPH"
    },
    "infixes":[
        "ch::Integer=1",
        "tr::Integer=1"
    ],
    "doc":"Hey"
}

We see here that the values key is saying that we are only going to accept Symbol type for our setindex! method and the symbol has to come out of symbols, a dictionary that is defined on the next line. The keys of this dictionary are going to be interpreted as symbols (e.g. :LogMagnitude) and the values are just ASCII strings to be sent to the instrument. We want to associate these symbols with the specific ASCII strings because these strings are not very descriptive, so we would like a more descriptive handle for them, as well as a handle that could be potentially shared between different instruments which have different "spellings" for the same command. We make the handles symbols because they are a more flexible type (which can always be parsed into strings)

generate_handlers makes a bidirectional mapping between the symbols and the strings. For the example above, the macro defines the following functions:

function symbols(ins::InsE5071C, ::Type{VNAFormat}, v::Symbol)
    if v == :LogMagnitude
      "MLOG"
    else
      if v == :Phase
        "PHAS"
      else...

        else
          error("unexpected input.")
    end
end

function VNAFormat(ins::InsE5071C, s::AbstractString)
    if s == "MLOG"
      :LogMagnitude
    else
      if s == "PHAS"
        :Phase
      else...


        else
          error("unexpected input.")
    end
end

The above functions will be defined in the module where the macro is run. Note that the function symbols has its name chosen based on the dictionary name in the JSON file. Since this function is not exported from the instrument's module there should be few namespace worries and we maintain future flexibliity.

source

# InstrumentControl.@generate_configure — Macro.

@generate_configure(instype, p)

This macro overloads the base method setindex!. In this implementation of setindex!, the instrument acts as the collection, the key is the InstrumentProperty type defined from the :type field of the p dictionary, and the value is specified by the user. This method also takes infix arguments; infixes are currently implemented as keyword arguments, so the setindex! method applies some standard infixes if none are specified by the user.

The method constructs a configuration command to send to the instrument to change the specific instrument property the dictionary p corresponds to (with user specified infixes). The command has "#" in place of the (user specified) value the property will be set to. It then sends the command to the instrument with the write method (NOTE: currently only defined in the VISA module for VISA instruments) where the command and the user-specified property value are passed to it. The write method replaces "#" with the proper value, and sends the command to the instrument

The macro accomplishes this by constructing and evaluating (approximately) the following expression:

function setindex!(ins::instype, v::values_Type, ::Type{p[:type]}, infixes_keyword_args)
  command=p[:cmd]
  command*" #"
  cmd = replace(cmd, "infix_name", infix_keyword_arg1)
  cmd = replace(cmd, "infix_name", infix_keyword_arg2)
  ... #etc
  ...manipulation of input v into format instrument accepts
  write(ins,cmd,v)
end

The function should be defined in the module where the instrument type was defined.

source

# InstrumentControl.@generate_inspect — Macro.

@generate_inspect(instype, p)

This macro overloads the base method getindex. In this implementation of getindex, the instrument acts as the collection, and the key is the InstrumentProperty type defined from the :type field of the p dictionary. This method also takes infix arguments; infixes are currently implemented as keyword arguments, so the getindex method applies some standard infixes if none are specified by the user.

The method constructs a query command to send to the instrument regarding the specific instrument property the dictionary p corresponds to. It then sends the command to the instrument with the ask method (NOTE: currently defined in the VISA module for VISA instruments). The ask method returns the value or state of the instrument property being queried.

The macro accomplishes this by constructing and evaluating (approximately) the following expression:

function getindex(ins::instype, ::Type{p[:type]}, infixes_keyword_args)
  command=p[:cmd]
  command[end] != '?' && (command *= "?")
  cmd = replace(cmd, "infix_name", infix_keyword_arg1)
  cmd = replace(cmd, "infix_name", infix_keyword_arg2)
  ... #etc
  ask(ins,cmd)
  ...further manipulation of output for display
end

The function should be defined in the module where the instrument type was defined.

source

Sweep, Sweep Jobs, and Sweep Queueing implementation

We stratify InstrumentControl "sweeps" functionality into different types, along with helper functions for each type, in order to achieve a object-oriented architecture with code modularity.

Measurement specific information, such as what independent variables will be swept and what response will be measured, are contained in a Sweep type:

# InstrumentControl.Sweep — Type.

mutable struct Sweep
    dep::Response
    indep::Tuple{Tuple{Stimulus, AbstractVector}}
    result::AxisArray
    Sweep(a,b) = new(a,b)
    Sweep(a,b,c) = new(a,b,c)
end

Object representing a sweep; which will contain information on stimuli sent to the instruments, information on what kind of response we will be measuring, and the numerical data obtained from the measurement. dep (short for dependent) is a Response that will be measured. indep is a tuple of Stimulus objects and the values they will be sourced over. result is the result array of the sweep, which need not be provided at the time the Sweep object is created.

source

However, additional metadata is needed for scheduling and queueing of sweeps, as well as logging of job information on ICDataServer. We "bundle" that information, along with a Sweep object, in a more comprehensive SweepJob type:

InstrumentControl.SweepJob
InstrumentControl.SweepJob()

Finally, we require a collection object that can hold SweepJob objects, and sort them by job priority, in addition to having functionality for automatic scheduling of jobs in the background. For this purpose we define the SweepJobQueue type, as well as a initialization inner constructor.

# InstrumentControl.SweepJobQueue — Type.

mutable struct SweepJobQueue
    #PriorityQueue is essentially a glorified dictionary with built-in functionality
    #for sorting of keys
    q::PriorityQueue{Int,SweepJob,
        Base.Order.ReverseOrdering{Base.Order.ForwardOrdering}}
    running_id::Channel{Int}
    last_finished_id::Channel{Int}
    trystart::Condition #used to communicate with the job_starter function
    update_taskref::Ref{Task} #used to communicate with the job_updater function
    update_channel::Channel{SweepJob} #channel for communicating with the job_updater function
    function SweepJobQueue()
        sjq = new(PriorityQueue(Int[],SweepJob[],Base.Order.Reverse),
            Channel{Int}(1), Channel{Int}(1), Condition())
        put!(sjq.running_id, -1)
        put!(sjq.last_finished_id,-1)
        sjq.update_taskref = Ref{Task}() #initializing a pointer for a task
        sjq.update_channel = Channel(t->job_updater(sjq, t);
            ctype = SweepJob, taskref=sjq.update_taskref)
        #the job_updater function is wrapped in a Task through this Channel constructor
        @schedule job_starter(sjq) #the job_started function is wrapped in a Task here
        sjq
    end
end

A queue responsible for prioritizing sweeps and executing them accordingly. The queue holds SweepJob objects, and "indexes" them by their job_id. It prioritizes jobs based on their priorities; for equal priority values, the job submitted earlier takes precedent. The queue keeps track of which job is running (if any) by its running_id Channel. The queue keeps track of the last finished job by the last_finished_id channel, for easy access to the data of the last finished job. Other fields are used for intertask communication. Note that a running_id of -1 signifies that no job is running.

When a SweepJobQueue is created through it's argumentless inner constructor (made for initialization purposes), two tasks are initialized. One task manages job updates, the other task is responsible for starting jobs; the former task executes the job_updater function, the latter task executes the job_starter function. Both functions execute infinite while loops, therefore they never end.

When the job starter task is notified with the trystart::Condition object in the SweepJobQueue object, it will find the highest priority job in the queue. Then, if a job is not running, and the prioritized job is waiting, and if the prioritized job is runnable (the priority may be "NEVER"), then the job is started. The database is updated asynchronously to reflect the new job, the queue's running_id is changed to the job's id, and the job's status is changed to "Running".

The job updater task tries to take a SweepJob from update_channel, the unbuffered job Channel of the SweepJobQueue object. The task is blocked until a job is put into the channel. Once a job arrives, provided the job has finished or has been aborted, the database is asynchronously updated, the job is marked as no longer running (by updating the running_id and last_finished_id channels), the sweep result is asynchronously saved to disk, and finally the job starter task is notified through the queue's trystart Condition object. The job updater task loops around and waits for another job to arrive at its channel.

source

# InstrumentControl.job_updater — Function.

job_updater(sjq::SweepJobQueue, update_channel::Channel{SweepJob})

Used when a sweep job finishes; archieves the result of the finished sweep job, updates all job and queue metadata, asynchronously updates ICDataServer, and notifies the job_starter task to run through sjq's trystart Condition object. This function continuously runs continuously without stopping once called. In its current implementation, this function is executed through a Task when a SweepJobQueue object is initialized; this allows the function to be stopped and recontinued asynchronously as is appropriate.

The function first waits until the update_channel is populated with a job. Once a job arrives, the function takes the job from the channel, and given that it's status is "Done" or "Aborted", it executes the items described above

source

# InstrumentControl.job_starter — Function.

job_starter(sjq::SweepJobQueue)

Used when starting a new sweep job. The function first obtains the highest priority job in sjq; and given that a job is not running, it's status is "Waiting", and if the prioritized job is runnable (the priority may be "never"), then the job is started. The function updates all job and queue metadata and asynchronously updates ICDataServer. This function continuously runs continuously without stopping once called. In its current implementation, this function is executed through a Task when a SweepJobQueue object is initialized; this allows the function to be stopped and recontinued asynchronously as is appropriate.

The last thing the function does is change the job status to "Running". When a job is scheduled with the sweep function, the sweep function waits for the status of the job to be changed from "Waiting" to start sourcing the instruments and performing measurements

source

When InstrumentControl is imported, a default 'SweepJobQueue' object is instantiated via the SweepJobQueue() constructor, and associated with a pointer called sweepjobqueue. This is THE queue running in the background automatically scheduling jobs, and referred to as the "default sweep job queue object" in the documentation. This object can be returned by the following function:

# InstrumentControl.jobs — Method.

jobs()

Returns the default SweepJobQueue object, initialized when the InstrumentControl module is imported/used. All jobs are scheduled in this object. Typically you call this to see what jobs are waiting, aborted, or finished, and what job is running.

source

Sweeps are scheduled by a call to the sweep function:

# InstrumentControl.sweep — Function.

sweep{N}(dep::Response, indep::Vararg{Tuple{Stimulus, AbstractVector}, N};
    priority = NORMAL)

sweep measures a response as a function of an arbitrary number of stimuli, sourced over the values given in the AbstractVector input, and returns a handle to the sweep job. This can be used to access the results while the sweep is being measured.

This function is responsible for 1) initialzing sockets for communication with the ICDataServer, 2) initializing an appropriate array to hold the reults of the sweep,

preparing a InstrumentControl.SweepJob object with an appropriate job_id

obtained from the database, 4) adding the job to the default SweepJobQueue object (defined when the InstrumentControl module is used/imported), and 5) launching an asynchronous sweep job.

The priority keyword may be LOW, NORMAL, or HIGH, or any integer greater than or equal to zero.

The actual sweeping, i.e., the actual source and measure loops to measure data are in a private function InstrumentControl._sweep!. Note that if InstrumentControl._sweep! is not defined yet, this function will also define the method in order to it to be used to schedule a sweep.

source

# InstrumentControl._sweep! — Function.

_sweep!(::Val{D}, ::Val{N}, sj, update_channel)

This is a private function which should not be called directly by the user. It is launched asynchronously by sweep. The implementation uses macros from Base.Cartesian. The stimuli are sourced only when they need to be, at the start of each for loop level.

D is the dimension of the output array of the measure function (if multiple things are measured for one Response type, the array will be multi-dimensional). N is the number of stimuli which the sweep sources over. sj is the handle to the SweepJob object, and update_channel is a channel of the queue used for intertask comunication.

The axis scaling of measure(sj.sweep.dep), i.e., the dimensions of the output array of the function. is presumed to be fixed, as it is only looked at once, the first time measure is called.

source