ROBOTICS
ABSTRACT
Our main task was to develop a device that measures mVenus fluorescence and adds liquids to sample containers.
Therefore, our team decided to build a fully automatized pipetting robot that is able to locate a set of samples, detect potential light emission and pipet a specific amount of non-natural amino acid solution into the fluorescent sample.
The foundation for the robot is a 3D-printer, due to the easy handling of movements in three dimensions. By controlling these movements with an optical system, the autonomy of the robot is further increased.
INTRODUCTION
Development of 3D Printers & Possibilities
In the 1980s, Chuck Hull invented the first standardized 3D printer, based on a procedure which is known as stereolithography (SLA, [1]). Moving from SLA to full deposit modeling (FDM) techniques, the 3D printing idea became alive in the do‑it‑yourself community. Ever since that time, simple 3D printers are accessible for little money and due to the open source idea of projects like REPRAP [2] affordable for many. In last years project, iGEM TU Darmstadt has already built a fully working SLA printer, capable of being fed with biologically manufactured plastics [3].
This year, the robotics team decided to rebuild a clone of the Ultimaker 2 FDM printer [4] and exchange the extruder with a camera and a pipet to create a pipetting robot. Using several open‑source parts and software, it is the idea to establish an easy-to-handle robot to assist the daily biologist's work.
iGEM TU DARMSTADT is a young and dynamic team of interdisciplinary and motivated researchers. Our advantage is, that we can bring together synthetic biology and classic engineering sciences, for which TU Darmstadt is famous. We have the possibility, thanks to iGEM, to experiment on our own ideas and to reach for the stars. Being interested in a variety of scientific topics, we wanted to mix up different talents to create a unique project.
GOALS
The main task is to develop a machine which is capable to monitor our organisms and their health condition (encoded by fluorescence) in order to keep them alive. Therefore the machine has to measure the light emission of the organisms and needs to be able to drop liquids into sample containers. This has to be independent of the exact position of the container, which requires an automatic tracking system.
The idea is that one places a container somewhere under the robot's working area and clicks a run button of a program. The robot starts its routine by tracking the new container and measuring the light emission of the organisms. Based on this measurement, the robot decides whether to feed the organisms with non‑natural amino acid solution or not. After a period of time it repeats this routine until the stop button of the program is clicked.
These are only the minimum requirements for our project's needs. We decided to go one step further and designed our robot in such a way, that it serves as a multi‑purpose platform which is adaptable and easy to modify. The open‑source character invites other scientists to add new features or improve the robot and its capabilities.
For example our dispensing system can be upgraded to be able to prepare 96-well plates with samples and monitor routines by using the optical system.
Additionally, our measuring head can be changed back to a printer head which allows to 3D print with just a few changes.
There is a vast room of possibilities, just using the concept of the accurate positioning of a sample in the 3D space.
Due to the fact that we try to stick to widely used open-source software and standard commercial parts, our machine can be easily combined with the most DIY products, making it reusable, flexible and cheap.
In the special case of the TU Darmstadt and the next generations of iGEM competitors, we had the idea to develop our technical equipment from year to year and, if possible, combine them. Our SLA printer from last year’s competition was upgraded and is nearly ready to use again, giving us the possibility to manufacture parts for prototyping in our lab. Also this year’s project will serve as starting point for the next year’s technical development team. New ideas and possibilities have been already discussed and we are looking forward to the next year’s competition.
SETUP OVERVIEW
FUNCTIONALITY
To fulfil the task of keeping the bacteria alive it loops through a specially designed procedure. Initially, the robot scans the working area for samples by illuminating the downside of the sample stage using infrared LEDs and monitoring the shadows of the placed reservoirs with a camera. If the contrast is sufficiently high it is able to detect the edges of the mentioned reservoirs, fit a circle onto it and compute the distance between the reservoir and the camera itself. Furthermore it is possible to put an entire rack of reservoirs under observation due to its ability to locate every individual reservoir.
Shortly after the detection, the distance information is sent to the 3D control program and the head of the robot moves in direction of the first reservoir. To check whether the bacteria needs more non‑natural amino acid the robot uses the fluorescence of the protein mVenus that is expressed by the bacteria. Therefore the robot excites the protein via high power LEDs and detects the emitted light. To exclude reflected light from the LEDs that would interfere with the measurement a longpass filter cuts off the spectrum below the emission peak of the protein. In dependence of the fluorescence signal, the robot decides whether it is necessary to pipet non‑natural amino acid onto the sample. If that is the case the robot moves the samples in z-range just so that the syringe reaches the sample and is able to securely add the non‑natural amino acid.
Eventually the robot recommences the procedure described above, except for the scanning of the individual positions of the samples, which are saved temporarily until all samples are checked. As long as the robot is activated, connected to a power supply and the syringe pump does not run out on non‑natural amino acid, the robot will loop through this whole process and keep the bacteria alive without the need of a human. Nevertheless it is possible to check what the robot is doing via a livestream of the camera visible on a graphical user interface, since there is no other opportunity to look inside the robot itself while it is working.
ACHIEVEMENTS
- Successfully redesign a 3D printer chassis to meet our requirements
- Construct a unique lightbox with integrated IR LEDs for positioning purposes
- Design a measuring probe with a camera device with an integrated optical filter system and LEDs
- Implementing an automatic object tracking system including a vector based feedback system for positioning
- Construct a syringe pump system to add liquids down to microliter accuracy
- Connecting a Raspberry Pi with an Arduino microcontroller by establishing a serial connection between the two devices, allowing a variety of different tasks
- Data of all CADs designed by the TU Darmstadt technical department
- A complete construction tutorial including a BOM (Bill of Materials incl. prices)
RESULTS
Circle Detection
The sample detection was facing two problems which on the first glance seem rather simple. Detect a circle and detect a rectangle. The detection of a circle is more easy due to the fact that it is a analytical function. This was already implemented by openCV and we were able to use the circle detection for our sample tracking system.
The green circles are the found objects. The red lines are vectors from the central position of our probe to the sample positions. They are used to drive the stepper motors so that our probe can move to them. You can also see, that changing the probes position are recognized and the vectors are recalculated.
Multi-Object Detection
A more delicate task was to detect round samples, stored in a rack. The difficulty is that the objects are intersecting and the algorithm needs to distinguish them somehow. You can see how the camera sees the whole object. Even with our eye it is difficult to see a rectangle. We improved the algorithm step by step using prominent features of the rack. The result is shown in the last picture.
Rectangle Detection
A rectangle detection is more complicated, but we were able to solve this challenge. You can see how the algorithm detect the rectangles and mark them, also tracking the single samples.
Full Functionality
As shown in the video our robot starts with its routine and going from sample to sample, checking for light emission. For demonstration purposes we installed a LED light. One can see, that the optic detect the LED light and then moves down to dispense liquid into the sample, because our pipette is installed sideways on the optical head. Then its moving back to its original position and continues its routine. At the end of the video we show the possible adjustments like brightness and focus.
In this video we additionally demonstrate the recognition of rectangles and circles. Using OpenCV and the Raspberry Pi Cam we can detect our glowing samples, here simulated with a green LED. After the detection of the glowing, in the case of a fluorescent protein induced by the blue high power LEDs, we pipet a drop of the respective substance (in our case nnAA) into the sample.
FURTHER DEVELOPMENTS
Due to a tight time schedule from the start to the end of iGEM it was not possible for us to realize all ideas and planned developments in respect of improvement of the robot itself.
For a working process with more kinds of bacteria cultures it is absolutely indispensable to develop a system that is able to avoid all sorts of contamination between the different bacteria. Therefore it would be an option to have an extra reservoir filled with ethanol in which the tip of the syringe can be sterilized between the checks of different samples.
Another modification that would be useful for working with individual bacteria cultures is making the power LED's changeable. This is necessary if the the wavelength of the LEDs does not overlap with the absorption spectrum of the fluorescent proteins or overlaps with a part of the spectrum that has a very low absorption efficiency.
Moreover, apart from the latter developments it may be useful to improve the syringe pump system. Instead of using a syringe pump it would be useful to use a system with a reservoir of liquids and a pump that works continously like a turbine, for example see http://www.ardulink.org/automatic-lipid-dispensing/.
Another useful modification of the robot would be to rebuild its foundation, namely an Ultimaker 3D-printer setup. Essential alterations would be to replace the sample stage with a heatbed and to replace the current head with a printhead hotend. Since the current head can be clipped it would not be too much of a challenge. Furthermore, a change of the syringe extruder is necessary, if the printer should work with plastics.
An alternative approach is a kind of paste 3D-printer. In this case it wouldn't even be needed to change the head and the syringe, because of the already viscous properties of the paste.
Optics
Operating Range of Wavelengths
Fluorescence Measurement and Filtering
Optical Hardware - Camera Head
We mounted the self-made autofocus in a cleanroom at "Institut für Druckmaschinen und Druckverfahren" TU Darmstadt. This was to keep the lense dustless. We are thankful to experiment under this conditions, but for the autofocus it is not mandatory. Be just aware of a clean dustless environment!
Optical Hardware - Lightbox
Cooling
Software
Marlin
There, you will also find a link where you can download the original, unmodified version of the marlin firmware.
OpenCV
OpenCV is a cross-platform image processing library and free for use under the open-source BSD license. Its development has been initiated by Intel in the nineties to demonstrate the capability of CPUs in executing complex image processing tasks. OpenCV covers the most basic morphological image operations up to advanced machine learning algorithms. It is written in C++, which is also its primary interface. By now most of the available features have been wrapped for other programming languages like Python, which we are going to use. Especially what we are looking for in OpenCV are its feature extraction algorithms, like the Hough transformation (1) (show details), contour and edge detection (2) (show details), and image moment (3) (show details) extraction. Also we make our lives easier by utilizing its implemented and optimized morphological operators and image filters (4) (show details), like the median filter and the sobel operator.
To enhance the long-term reliability of detection we separate the detection procedure into two steps: Since racks are predominantly used to hold the sample vessels, we firstly extract and detect the rectangular shaped racks with contour and thresholding techniques. Several image shape descriptors are used to track possible positional changes. The user is then allowed to manually assign regional attributes (e.g. radii, heights) for each rack. The assignment of local radii helps the circle detection in the second step. Not only does it inhibit false detections, but also allows for a proper multithreaded apportionment of work.
PyQt
Qt is a software tool to develop a GUI (Graphical User Interface)[7]. It is available under a commercial and an open-source license. The software is a cross-platform application framework, which means it runs on the most computer systems like Unix or Windows. The underlying programming language is C++ and Qt can use already existing programming languages like Javascript, making it a powerful tool.
The main idea of Qt is to use a system of signals and slots to have an easy framework to connect displayed elements with underlying functions. Also, the reusability of already existing code is enhanced. Every graphical element, for example a button, emits its own signal when it is pressed or used. The signal then can be used to trigger an action, like closing a window. If the signal is not connected to a function nothing will happen, however the signal will be emitted with no consequences. Now it is possible to connect the emitted signal with a desired action, called slot, and the program gets its specific behavior.
Qt is widely used by companies like the European Space Agency (ESA), Samsung, DreamWorks, Volvo and many more.
To be able to combine the possibilities of Qt with the simplicity of the Python programming language, PyQt was developed. PyQt is a binding for Python capable of translating Qt methods within the Python syntax.
To be able to get a direct preview of the constructed GUI, Qt Designer is a helpful tool. Basically it enables an intuitive way to build a graphical user interface without a need to explicitly coding it. To later work with the code itself, PyQt uses a method called pyuic(number), which is executed through the terminal. The number in the brackets stands for the version.
After converting the code one can open the GUI as a regular python script and work with it as usual.
BUILDING INSTRUCTIONS
Construction Video
Circuit Diagram
The stepper motors for X and Y are plugged into the RAMPS board as shown in the graphic. There are already prepared cables available with corresponding connectors. To control and power them the stepper driver (A4988) are plugged in the RAMPS slots, marked with X and Y. On the bottom of the stepper driver it is marked which pin is ground and so on. Also on the RAMPS board it is marked where the corresponding slots for ground etc. are. The X and Y stepper motors have one endstop each to provide a stop signal for the homing routine. It is also possible for safety reasons to add one more endstop to each axis, so that the movement stops for sure at the end positons. All the endstops are plugged into the marked position on the RAMPS board. From left to right the pins are for X min and X max position, Y min and Y max position and Z min and Z max position. The pin marked with a + is reserved for the 5 V, the middle pin is the ground and the last pin marked with a S is the signal.
The Z stepper motor has two endstops and is connected like the X and Y stepper motors and endstops.
To drive our syringe pump one more stepper motor and one endstop is required and is plugged into the E0 position. It is connected like X,Y and Z.
To power the RAMPS board 12 V are needed, for example from an ATX power supply.
The Raspberry Pi3 is responsible for controlling the optics and LEDs whereas the Arduino controls the stepper motors and endstops. Arduino and Raspberry Pi are connected via one of the Pi’s USB port and the USB Type B input of the Arduino.
The lightbox is powered with 24 V from the ATX power supply. The lightbox consists of 4 LED bars, each carrying 10 infrared LEDs in series. Each bar is driven by one 50mA constant current source. These are parallel connected to the -12V of the power supply, and to the NC of one Relay. The COM port of that Relais is then connected to the +12V of the power supply.
The 2 Relay Modul serves as a Raspberry Pi controlled switch which controls the lightbox and the blue exposing LEDs. The corresponding wire’s number refer to the used GPIO pins. Be careful, the internal GPIO layout is different as the numbering on the Pi layout scheme. All numbers correspond to the latter!
The L298N are DC motor drivers which we alienated as protection circuit, to separate the 12 V from the Raspberry Pi’s GPIO.
The blue exposing LEDs are series-connected, and controlled by the 2 Relay Modul on the first side.
Required Software
In case you want to rebuild ou project, you can download all needed software in this section. The software includes the exact marlin firmware, which we uploaded on the Arduino, as well as the software that is installed on the Raspberry Pi and some improved python scripts. Furthermore, to enable wireless support, install and configure "hostapd" and "dnsmasq" on your Raspberry Pi. We followed this external tutorial.
Bill Of Materials
References
- [1] http://edition.cnn.com/2014/02/13/tech/innovation/the-night-i-invented-3d-printing-chuck-hall
- [2] https://2015.igem.org/Team:TU_Darmstadt/Project/Tech
- [3] http://reprap.org/wiki/About
- [4] http://www.thingiverse.com/thing:811271, jasonatepaint
- [5] https://picamera.readthedocs.io/en/release-1.12/
- [6] https://www.plexiglas-shop.com/pdfs/en/212-15-PLEXIGLAS-LED-edge-lighting-en.pdf
- [7] https://en.wikipedia.org/wiki/Qt_(software)