Team:UCSC/Software

Software Overview

With the power of Python and a team of five, the bioinformatics team: designed a way to parse status files (a JSON created by the bioreactor) into a database, built a server, built the framework for a website with multiple pages and tabs, and built methods that pull specific history of the bioreactor when requested: this is especially important for data visualization.

Our plan to create a networked, scalable bioreactor system began with realizing the collaboration of two efforts: a hardware side installation which could manage the bioreactor autonomously, and a server installation in the cloud to host bioreactor data sent to it and serve as an internet accessible console to control and monitor system conditions from a smartphone.

Github Links for Each Project:

Raspberry Pi Software: https://github.com/UCSC-iGEM-2016/taris_controller
Cloud Server Software: https://github.com/UCSC-iGEM-2016/taris_server


Simplified Flowchart:

Mountain View

Cloud Server Software: Taris Server



Goals for our server interface:

  • Simple, User-Friendly Website GUI
  • SQL Database to Store Data History
  • Security (User Accounts/Passwords for Control)
  • Beautiful Data Visualization
  • Receive and Send Data to the Raspberry Pi

File Structure of Taris Server:

  • __init__.py (Blank; Required to make Python treat the directories as containing packages.)
  • TarisV1.py (Flask-based server; updates graphics with data; sends and receives JSON packets to/from the pi; controls the website.)
  • setupDB.py (Creates a database Bioreactor.db using SQL Alchemy.)
  • console.html (Remote control interface for the bioreactor.)
  • index.html (Main page with updated data graphics for all current gauge readings.)
  • plots.html (Plots of all sensors (last 5 minutes + custom timeframe).)
  • plotsMotor.html (Plots of motor current (last 5 minutes + custom timeframe).)
  • plotsPH.html (Plot of pH sensor readings (last 5 minutes + custom timeframe).)
  • plotsPHBokeh.html (Plots of pH sensor readings using Bokeh (last 5 minutes + custom timeframe).)
  • tempPlotBokeh.html (Plots of temperature sensor readings using Bokeh (last 5 minutes + custom timeframe).)
  • tempPlots.html (Plots of temperature sensor readings (last 5 minutes + custom timeframe).)
  • testParams.html (Temporary file.)
  • menu_style.css (Style sheet for our menu.)


DESIGN OVERVIEW


We've created a flask server with python that can control a bioreactor and visualize the data generated by its sensors/motors in a simple, easy-to-understand format.
Flask is the server framework, SQL Alchemy is the database we use to store our data so that we can recall history or analyze our data over time, the actual webpages use HTML5 so that they are smartphone accessible, and we make use of a css stylesheet for greater control of HTML visuals. Javascript is borrowed from Raphaël's JavaScript Vector Library (raphael-2.1.4.min.js; MIT License) and Bojan Djuricic's animated gauges (justgage.js; MIT License). JSON packets allow for sending parameter data to the raspberry pi to update and control motor function, with a user security system that password protects this control while leaving the main body of the website accessible to the general public. JSON packets are also received in order to update the visual graphics on the website (gauges and line plots) in the form of: temperature sensor data, motor current readings, and system parameters.

SUMMER EXPERIENCE

The first five weeks of our programming was essentially just learning. We figured that we needed a server to send data to and a website for the team to use (change the bioreactor values, see how it was doing, etc.). It took us about a week to figure out all the components necessary to accomplish these two things. There were six components to this, we spent the first five weeks learning about them: first, what a server backend looks like, second, how to connect to a server (another computer), third, how to build a database, fourth, how to do basic HTML5 (mobile compatible) web pages, fifth, how to use Bokeh, a data visualization module in Python, sixth, how to let the user request specific information from the server: either causing the bioreactor to change values or updating a graph to a specific time range. By the end of the first five weeks we had a framework of a website, but looking back, there were so many pieces still to add (not to mention the fine tuning).

To accomplish what we have done, each member was given a task to focus on. At first there were three tasks: build the database structure, build the website framework (how does it look), and build the history graphs. This parallel programming worked exceptionally well and by the end of the first five weeks the bioinformatics team felt ready to launch a site. Little did we know at the time that we still had far to go.

The last step was to acquire a domain to launch our server. We had three options when we choose where the website would be hosted. Launch it from one of our computers and always have it on, Amazon Web Services, or the school’s network of virtual machines (VM). Each had advantages and disadvantages but I will only discuss why we went with the school’s network. We choose to use the local network at UCSC to host our server for three key reasons. The first being we were able to get on site support, a simple walk over to Jack Baskin Engineering yielded three open doors, willing to help on the spot. Second, the team that manages these networks also provides free guidance around how to protect the VM; such as a firewall or preventing false data injection (SQL-injection in our case). Third, it was free for the size we needed. Further, but not the reason we choose it, we were able to get the domain we wanted! These reasons led us to launch a site you can go to anytime and anywhere (provided internet access) at sugarslugsbr.soe.ucsc.edu.

The website was not great at the end of the first five weeks and so we had plans to add key features that would make the site usable, reliable, and timely. We were able to implement three main abilities into our site that make it useful to our team and a good template for the future: one, the ability to receive data, in our case from a Raspberry Pi™ and store it in a database; two, set the desired temperature and pH of the reactor (with a password); third, visualize the data either the default of the last five minutes or in a time span the user specifies. These three methods allow for two way communication of the bioreactor and the server as well as visualization of how the bioreactor is responding to user inputs.

CHALLENGES



1. JSON vs. Pickles.
Data structures can be difficult. JSON is a format that is used across serveral programming languages while pickles are Python specific. While useful in some cases, pickles were removed from our code.
2. Apache, Ubuntu, and Installation
Installation can be easy or hard. If you have sudo access, it is easy. If you do not know what sudo is, it will probably be confusing. Luckily the UCSC School of Engineering (SOE) has a wonderful virtual machine (VM) department. Tim Gustafson took time out of his schedule to have a meeting with us, get set up, and was available for questions following the launch of our VM.
3. Slow to update data (history functionality)
Interestingly, when there is constant data being sent to the server it preforms well. However, the site may appear to be down often because we are not constantly using the bioreactor. When data is flowing, the server seems happy and we were able to speed up the loading time considerably.
4. Getting the server up and running (cloud; AWS; SOE)
As mentioned previously, there is a VM department at our school. Although AWS is used more often in industry, the SOE was on our campus and easy to get access to. Another advantage, because we are not dealing with huge amounts of data or requests, the SOE VM is free for us to use.
5. Two vs. One SQL database
Consolidating from two to one database helped speed up our server. It was also realized that two databases are not often used; one database will often contain many tables. This was a great learning experience for our team. We learned that diving deep is often the best way to go. Do your homework/research and you will get results.
6. Bokeh vs. D3
Bokeh is a simple tool that makes online integration easy. D3 is the industry standard for data visualization. We choose to go with Bokeh because we had templates of it built in by the time we learned about D3. Further, we worked on improving the functionality over looks: we didn't waste time.
7. Server Not Loading. Restart instructions via manual.
Flask is a lightweight server, according to the Flask homepage. For our purposes it generally works and runs quickly. However, after weeks of being constantly on, it slows down. Our team became very familiar with how to launch (restart) the server because we often needed to.


RESULT (sugarslugsbr.soe.ucsc.edu)

Raspberry Pi Software: Taris Controller


Goals for our hardware interface:

  • Simple, User-Friendly UI
  • Interfacing/Reading Sensors (Using I2C Protocol)
  • Sending a PWM to Control Motors/Heaters
    • 4 Motors (Peristaltic Pumps): 12V, 300mA
      • Chemostat Inflow Motor
      • Chemostat Outflow Motor
      • NaOH Supply Motor
      • Backup Motor
    • 1 150W Heating Element (12"x12" Flat Water-Resistant Pad)
  • Adjust PWM in Response to Sensor Data
  • Receive and send data to a server

File Structure of Taris Controller:

  • __init__.py (Blank; Required to make Python treat the directories as containing packages.)
  • taris_adc.py (Functions to Convert Analog Sensor Data to Digital)
  • taris_calibrate.py (PID Adjustment Functions)
  • taris_hw.py (UI and Main Program)
  • taris_json.py (Network Handler Functions)
  • taris_motors.py (Motor Control Functions)
  • taris_reactor.py (Bioreactor Object and Extends UI)
  • taris_sensor.py (Sensor Data and Calibration Functions)


DESIGN OVERVIEW


We've created a python program for the raspberry pi that can control motors, as well as read sensor data on our bioreactor using the raspberry pi's IO pins. The bioreactor harbors a fully autonomous feedback system implemented to maintain ideal temperature and pH for bacteria/yeast. Our software currently records temperature sensor data, motor current readings, and system parameters. It then creates a JSON packet using this data, and sends it to our cloud server for live viewing.

Conversely, our software can be updated via the internet, and so checks the server routinely for new commmands given in server-created JSON packets. Our system was built to be easily scalable, and this was kept in mind so that we could potentially add more sensors and motors in the future.

CHALLENGES



1. Getting the Raspberry Pi to see the I2C buses.
Initally we had difficulty getting the I2C buses to show up on the raspberry pi and this was simply because we'd forgotten two pull-up resistors on the sensors. After adding this, everything worked.
2. Updating the server and eduroam.
Eduroam requires specific protocols to work with the raspberry pi, and the wpa_supplicant file must be modified to see the network. This was eventually fixed through tweeking common protocols posted for other other eduroam networks posted online.
3. Motors initially not working.
Transistors needed to be resoldered initially. This took considerable time and Henry Hinton did amazing work getting it working.
4. Design of the JSON file.
We debated for a few days over the structure of the JSON files to send, which looking back seems silly (it's basically a text file that looks like a python dictionary), but we had no idea what information we'd be sending and none of us were familiar with the format.
5. Characterizing the motors and NaOH control (small amounts slowly).
This took several days. Motors all vary and we wanted to get controlled amounts of NaOH with our cheap 10 dollar peristaltic pumps. We attempted this with acidified water as well as live culture, and eventually settled on a very slow addition rate as the culture does not acidify quickly.