Overview

The pcdshub python 3 software stack is being developed with the following design goals:

• Modules should have a well-defined, restrained scope.

• Code should be readable, reusable, and sharable.

• Each piece of data should have a canonical source for all applications.

• Deployment and testing should be automated.

• The open source community should be maximally leveraged.

These design goals result in a matrix of Github repositories and python modules that are individually understandable, but together obscure the bigger picture. The goal of this page is to explain at a high level how each application works and what each repository does.

About this Page

This page will explore the applications and modules from pcdshub, as well as some external modules, some of which are developed at SLAC and some which are not.

For each project, this page will explain:

• The motivation behind the project: what problem is being solved?

• How the project works at a high level

• What other projects does this one rely on?

• What other projects rely on this one?

Definitions

An application is any python code that runs stand-alone to accomplish a task. These may rely on modules to accomplish their goals. A module is code with an importable API and a means to install itself in a python environment. Many repositories are both applications and modules, providing both a stand-alone script and an API for other applications to leverage.

Applications

This section will list our applications and explain the motivations behind them, as well as their dependencies. This includes only repositories that are hosted on pcdshub and have a corresponding online documentation page.

Hutch-specific Repositories

Repository	amo, sxr, xpp, xcs, mfx, cxi, and mec
How we use it	We put softlinks like mfx3 in the hutch opr’s path. These link to scripts like mfxpython in the repository that launch a convenient IPython environment for controlling the hutch and automating DAQ data collection. The repository is used to host hutch-specific classes, devices, and configurations that may later be merged into a shared module.
Motivation	We need dedicated landing areas for hutch-specific and experiment-specific code to satisfy the needs of a constantly shifting environment. This can’t go into a module because the module can’t control which python environment you launch it from.
How it works	A version file e.g. mfxversion chooses a shared conda environment and may set up development package overrides. The mfxpython script enters this environment using hutch-python and a brief conf.yml configuration file. These coordinate the loading of shared services and provide hooks for the hutch-specific code.
Dependencies	These may vary between hutches, but all hutches rely on hutch-python as the launcher and configuration reader. All of these repositories were created from the hutch-python --create reponame template function.

hutch-python

Repository	hutch-python
How we use it	Scripts like mfxpython call on the hutch-python application with a hutch-specific conf.yaml file, as well as hutch-specific mfx/beamline.py and experiments folder.
Motivation	The load process for each hutch is essentially identical. This should be handled in shared code with simple hooks for hutch-specific and experiment-specific additions. This was chosen instead of supporting IPython profiles or massive beamline.py files because these can get messy and hard to manage. This avoids the procedure of going around to each deployment to update boilerplate code.
How it works	A shared startup script is seeded with the contents of the conf.yml file. Shared operations are done the same way and then the beamline.py and experiments user code is invoked.
Dependencies	This is used by the Hutch-specific Repositories to Relies on most of our modules: Uses happi to load from the beamline device database and from the questionnaire experiment devices Uses pcdsdevices for the device definitions and to set up position presets Uses pcdsdaq to control the data acquisition system Uses lightpath as a command-line interface for checking if anything is blocking the beam path Uses elog to post to the experiment logbooks

lightpath

Repository	lightpath
How we use it	An environment with lightpath installed has a lightpath script that will open the GUI. This can be use to visually inspect the light path. This doubles as an application and a module, providing an importable interface that is used in other places, such as hutch-python.
Motivation	Help the user find which objects are in the beam, and which are not. This can be used to clear the beamline and to check if you expect beam to be incident on your imager. Unlike older software with the same purpose, this is extremely configurable and simple to keep up-to-date using happi.
How it works	This uses the z-position and beamline metadata from happi to sort devices by position along the beamline. It relies on the in/out interface from pcdsdevices to determine whether each device is in the beam.
Dependencies	Uses pydm for the GUI Uses happi for device loading Uses modules like pcdsdevices to define device classes

pmgr

Repository	pmgr
How we use it	This is the Parameter Manager, pmgr for short. This isn’t installed in an environment (yet), but is used as a stand-alone GUI for keeping track of device parameters. It was used in the old python CLI for the same purpose, but this has yet to be ported.
Motivation	We need a place to keep track of and restore parameters to ease the deployment and redeployment of motors for changing experimental needs.
How it works	pmgr provides a qt gui with a mysql server backend that stores all of the parameters.
Dependencies	pmgr does not depend on our other modules, but it does depend on our patched version of pyqt to make the tables work.

pcds-envs

Repository	pcds-envs
How we use it	This is used to track our shared environment changes, and to anticipate and catch integration problems with bringing all of our packages together.
Motivation	We need to do this in a structured, semi-automated way to minimize upkeep and catch mistakes.
How it works	packages.txt is updated to add new packages. stage_release.sh is run to build a new environment from the packages list and push it to Github. We make a PR and discuss. Along with the PR, an automated test is run to check that all of our pcdshub modules pass their individual automated tests with the full environment (this also runs daily). If we like the new environment, we merge the PR and make a tag. We use apply_release.sh to put the new tag into the shared environment.
Dependencies	This only depends on Python and Conda

pcds-recipes

Repository	pcds-recipes
How we use it	This is used to build non-pcdshub Conda packages and make sure they get into the pcds-tag channel.
Motivation	It is unsustainable and unstable to rely on special channels for our conda environments. By limiting to the defaults, conda-forge, and our own pcds-tag, it becomes easy to specify our environment setups.
How it works	Recipes are placed into the repository and build.py builds and uploads them.
Dependencies	This only depends on Python and Conda

Modules

This section will list our modules and explain the motivations behind them, as well as their dependencies. This includes only repositories that are hosted on pcdshub with a corresponding online documentation page.

typhos

Repository	typhos
How we use it	To automatically generate screens from ophyd devices.
Motivation	We need new pydm screens and these are great starting points. The ophyd device structure is very useful for organizing device properties into logical groups.
How it works	Groups device components by kind (e.g. hinted, normal, configuration) and sort into tabs, etc. accordingly. Provides tools on every generated screen.
Dependencies	pydm and ophyd

pcdsdevices

Repository	pcdsdevices
How we use it	To define ophyd device classes that correspond to LCLS PCDS EPICS IOCs. This is also where we put additional logic and CLI niceties for interactive sessions with these devices.
Motivation	There must be a common place where these classes are defined so that all of our Python applications share PV structures and logic for each identical device class or instance.
How it works	Follows the ophyd rules to record which PVs are associated with each device class.
Dependencies	ophyd pyepics, currently as the EPICS communication layer

transfocate

Repository	transfocate
How we use it	To define an ophyd device class for the MFX transfocator.
Motivation	This is more complicated than a standard device and was being developed parallel to the pcdsdevices repository.
How it works	Defines the transfocator’s PVs and some helper methods for putting lens combinations in and out.
Dependencies	pcdsdevices

hxrsnd

Repository	hxrsnd
How we use it	To define ophyd device classes and bluesky scanning routines for the XCS split and delay.
Motivation	This is more complicated than a standard device and was being developed parallel to the pcdsdevices repository.
How it works	Defines the SND’s PVs and some helper methods for running calibration scans and collecting data.
Dependencies	pcdsdevices, pswalker

nabs

Repository	nabs
How we use it	To define lcls-specific bluesky utilities and to build out the bluesky automation framework.
Motivation	We want a cleaner version of the legacy pswalker module to have shared code between different applications that need to do similar things e.g. filter on beam drops, maximize signals, etc.
How it works	Defines an API for various plans, preprocessors, etc.
Dependencies	bluesky

pcdsdaq

Repository	pcdsdaq
How we use it	To define a bluesky-compatible control layer for the LCLS1 data acquisition system (DAQ).
Motivation	We need to be able to control the DAQ. Doing it in a bluesky-compatible way means we can include the DAQ in any bluesky plan.
How it works	Wraps the pydaq.Control object and adheres closely to the bluesky interface. You can configure the daq, do runs, and include it as a readable device in scans.
Dependencies	bluesky, ophyd, pydaq

happi

Repository	happi
How we use it	To store all of our device metadata in device_config and reload these devices to create identical objects in every application.
Motivation	There must be a canonical way to load devices, consistent across all of our python applications.
How it works	Defines container objects that represent real objects. These can be stored in json or mongodb and be used to generate real objects.
Dependencies	No pcdshub dependencies

device_config

Repository	device_config
How we use it	As a central file store of all the happi device configuration information.
Motivation	We need to store the information somewhere until we have a mongodb set up.
How it works	Stores a single json file.
Dependencies	No pcdshub dependencies

elog

Repository	elog
How we use it	To post information to the experiment elog.
Motivation	We need python bindings for this so that we can update the elog programmatically or from the command line.
How it works	Through https requests
Dependencies	No pcdshub dependencies

External Modules

This section will list some of the more site-specific external modules we use and explain the motivations behind the modules and why we use them.

bluesky

Repository	bluesky
How we use it	As the scanning, process flow, and automation platform.
Motivation	If we do scanning in a structured way, we can write cleaner code that can continue to be maintained, and we can take advantage of anything that the bluesky community has written as far as visualizing data and controlling process flow.
How it works	A central RunEngine object processes user-created generators called plans. These can do things like move motors, read values, etc., but they can also be inspected to see what the plan would do if we ran it, and they can be given arbitrary adaptive logic since this is Python.
Dependencies	No pcdshub dependencies

ophyd

Repository	ophyd
How we use it	As the device control abstraction layer.
Motivation	We need to have a sane convention for defining devices, and a consistent way to define motion interfaces, callbacks, etc.
How it works	Device subclasses register Component attributes so that every instance knows which PVs to connect to.
Dependencies	No pcdshub dependencies

pydm

Repository	pydm
How we use it	To create/display pyqt-based EPICS control screens.
Motivation	We need to be able to include Python logic in our screens for advanced applications, but we also need a way for new users to create screens quickly using a drag-and-drop interface.
How it works	This leverages qt designer as a drag-and-drop interface. A rich Python API is provided for interacting with EPICS PVs and archiver data, among other things. Widgets are provided that correspond to the legacy EDM widgets.
Dependencies	No pcdshub dependencies