ROBOTICS

ABSTRACT
Our main task was to develop a device that measures mVenus fluorescence and adds liquids to sample containers. Therefore, our team decided to build a fully automatized pipetting robot that is able to locate a set of samples, detect potential light emission and pipet a specific amount of non-natural amino acid solution into the fluorescent sample.
The foundation for the robot is a 3D-printer, due to the easy handling of movements in three dimensions. By controlling these movements with an optical system, the autonomy of the robot is further increased.

INTRODUCTION

Development of 3D Printers & Possibilities

In the 1980s, Chuck Hull invented the first standardized 3D printer, based on a procedure which is known as stereolithography (SLA, [1]). Moving from SLA to full deposit modeling (FDM) techniques, the 3D printing idea became alive in the do‑it‑yourself community. Ever since that time, simple 3D printers are accessible for little money and due to the open source idea of projects like REPRAP [2] affordable for many. In last years project, iGEM TU Darmstadt has already built a fully working SLA printer, capable of being fed with biologically manufactured plastics [3].
This year, the robotics team decided to rebuild a clone of the Ultimaker 2 FDM printer [4] and exchange the extruder with a camera and a pipet to create a pipetting robot. Using several open‑source parts and software, it is the idea to establish an easy-to-handle robot to assist the daily biologist's work.

iGEM TU DARMSTADT is a young and dynamic team of interdisciplinary and motivated researchers. Our advantage is, that we can bring together synthetic biology and classic engineering sciences, for which TU Darmstadt is famous. We have the possibility, thanks to iGEM, to experiment on our own ideas and to reach for the stars. Being interested in a variety of scientific topics, we wanted to mix up different talents to create a unique project.

References:

GOALS

The main task is to develop a machine which is capable to monitor our organisms and their health condition (encoded by fluorescence) in order to keep them alive. Therefore the machine has to measure the light emission of the organisms and needs to be able to drop liquids into sample containers. This has to be independent of the exact position of the container, which requires an automatic tracking system.
The idea is that one places a container somewhere under the robot's working area and clicks a run button of a program. The robot starts its routine by tracking the new container and measuring the light emission of the organisms. Based on this measurement, the robot decides whether to feed the organisms with non‑natural amino acid solution or not. After a period of time it repeats this routine until the stop button of the program is clicked.
These are only the minimum requirements for our project's needs. We decided to go one step further and designed our robot in such a way, that it serves as a multi‑purpose platform which is adaptable and easy to modify. The open‑source character invites other scientists to add new features or improve the robot and its capabilities.
For example our dispensing system can be upgraded to be able to prepare 96-well plates with samples and monitor routines by using the optical system. Additionally, our measuring head can be changed back to a printer head which allows to 3D print with just a few changes.
There is a vast room of possibilities, just using the concept of the accurate positioning of a sample in the 3D space.
Due to the fact that we try to stick to widely used open-source software and standard commercial parts, our machine can be easily combined with the most DIY products, making it reusable, flexible and cheap.
In the special case of the TU Darmstadt and the next generations of iGEM competitors, we had the idea to develop our technical equipment from year to year and, if possible, combine them. Our SLA printer from last year’s competition was upgraded and is nearly ready to use again, giving us the possibility to manufacture parts for prototyping in our lab. Also this year’s project will serve as starting point for the next year’s technical development team. New ideas and possibilities have been already discussed and we are looking forward to the next year’s competition.

SETUP OVERVIEW

To get short information about our robot's single parts, click on the part of interest.
To realize the movement of the head in the x-y-plane, the head is mounted onto a 3D-printed connection element which is fitted between two crossing 6 mm aluminum linear rods. To ensure a smooth sliding of the connection on the linear rods, two LM6UU linear bearings were applied. Also, the crossing 6 mm aluminum linear rods are clipped on four UM2 Ultimaker 2 Injection Sliding Blocks which are themselves sliding on 8 mm steel rods. Two F688ZZ flanged ball bearings per 8 mm steel rod are plugged into SK16 rod supports ensure that they are able to rotate with less friction. Furthermore, GT2 timing belts transfer the stepper motor's rotation into a linear movement of the head. These timing belts connect the stepper motors with the 8 mm steel rods with the usage of GT2 pulleys. Timing belts that are fixed in the sliding blocks range from one linear rod on one side of the robot to a parallel one on the opposing side. Therefore, an amount of two timing belts per axis and one timing belt per motor are necessary for a proper movement in the x‑y‑plane.
For the motion in z-direction, a threaded rod is coupled to a stepper motor with a 5‑to‑8 mm shaft coupling and vertically mounted with two KP08 pillow blocks. The rotation of the threaded rod is then transmitted into a vertical movement of the sample stage via a ball screw. For a greater stability and a more balanced force distribution, two additional 12 mm aluminum rods help to guide the vertical movement of the sample stage. Besides, one LM12LUU linear bearing per vertical axis is used for nearly frictionless motion and an even more balanced force effect on the vertical axis. The mounting for the latter is attached on the sample stage and particularly manufactured, again for a balanced force distribution. The sample stage itself consists of an aluminum framework with an infrared‑transparent Plexiglas. To minimize the weight of the sample plate, while keeping its stability, the thickness of the table's components was chosen to be 3 mm.
The main part of our optics is the combination of our lightbox, LEDs and a Pi NoIR v2 camera. With these parts we detect flourescent light, after stimulated emission. The localization works with the lightbox via built‑in infrared LEDs whose light spreads inside a diffusion plate, generating an evenly distributed light source. Emerging shadows can be detected with the camera. The latter is attached to the head of the robot that additionally consists of optical components such as four high power LEDs, various filters, a lens mounting and a tip that is connected to the syringe pump. As already mentioned, the camera is in charge of detecting different sample locations and fluorescent light of the mVenus protein. While the high power LEDs ensure the excitation of the protein; the filters, that are composed of a longpass-filter and sunglasses, eliminate reflected light such that the camera only detects relevant signals. The lens mounting works as an autofocus that is able to control the sharpness of the camera. This works by applying different voltages to coils that move the lens inside a modified webcam.
The syringe pump is the device of the robot that feeds the bacteria with non-natural amino acid if it is necessary. Its setup is a stepper motor that squeezes the non-natural amino acid out of the syringe and a 3D-printed framework that holds the syringe. It also includes a threaded rod for transmission between the stepper motor and the syringe. To keep the whole setup at a fixed position and still pipet on the right sample, the end of the syringe is connected with the head by a flexible tube.
The framework for the robot is built in a form of a simple cuboid constructed with 30 mm x 30 mm aluminum profiles. These profiles are connected with each other via M8 screws and drilled into ISO metric screw threads. Two extra horizontal aluminum profiles are added on the backside which hold the axes and the threaded rod needed for the vertical motion. Moreover, a thin aluminum frame is attached at the lower part of the 30 mm framework to mount the lightbox onto it.
The electrical parts used in this robot can be separated into those that directly manage the threedimensional movements and those that are in charge of the optical localization and detection. The threedimensional movements are controlled by an Arduino Mega 2560 microcontroller connected to a RepRap Arduino Mega Pololu Shield 1.4 (short: RAMPS 1.4) which provides an intuitive connection of drivers and other electronical devices without using wires. The drivers used in this project are called Pololu - DRV8825 Stepper Motor Driver Carrier and allow the Arduino to let the stepper motors make a 1/32 step and therefore increases the spatial resolution. Additional parts involved in threedimensional movement are the so called endstops that assure that the head does not crash into the boundaries of the robot. The list of parts that control the optical localization and detection are comprised of a single‑board computer Raspberry Pi 3 and controls several other electro‑optical components. All necessary power is delivered by a standard ATX power supply.

FUNCTIONALITY

The functionality of the pipetting robot comprises of the three‑dimensional agility of a 3D-printer and the possibility to pipet a specific amount of non‑natural amino acid using a syringe pump. Also it has intelligent visual object recognition so that it is able to distinguish between samples that require more non‑natural amino acid from samples that still contain a sufficient amount. With that said it is capable to autonomously keep alive the modified E. coli bacteria, given that it is activated and connected to a reliable power supply.
To fulfil the task of keeping the bacteria alive it loops through a specially designed procedure. Initially, the robot scans the working area for samples by illuminating the downside of the sample stage using infrared LEDs and monitoring the shadows of the placed reservoirs with a camera. If the contrast is sufficiently high it is able to detect the edges of the mentioned reservoirs, fit a circle onto it and compute the distance between the reservoir and the camera itself. Furthermore it is possible to put an entire rack of reservoirs under observation due to its ability to locate every individual reservoir.
Shortly after the detection, the distance information is sent to the 3D control program and the head of the robot moves in direction of the first reservoir. To check whether the bacteria needs more non‑natural amino acid the robot uses the fluorescence of the protein mVenus that is expressed by the bacteria. Therefore the robot excites the protein via high power LEDs and detects the emitted light. To exclude reflected light from the LEDs that would interfere with the measurement a longpass filter cuts off the spectrum below the emission peak of the protein. In dependence of the fluorescence signal, the robot decides whether it is necessary to pipet non‑natural amino acid onto the sample. If that is the case the robot moves the samples in z-range just so that the syringe reaches the sample and is able to securely add the non‑natural amino acid.
Eventually the robot recommences the procedure described above, except for the scanning of the individual positions of the samples, which are saved temporarily until all samples are checked. As long as the robot is activated, connected to a power supply and the syringe pump does not run out on non‑natural amino acid, the robot will loop through this whole process and keep the bacteria alive without the need of a human. Nevertheless it is possible to check what the robot is doing via a livestream of the camera visible on a graphical user interface, since there is no other opportunity to look inside the robot itself while it is working.

ACHIEVEMENTS

Successfully redesign a 3D printer chassis to meet our requirements
Construct a unique lightbox with integrated IR LEDs for positioning purposes
Design a measuring probe with a camera device with an integrated optical filter system and LEDs
Implementing an automatic object tracking system including a vector based feedback system for positioning
Construct a syringe pump system to add liquids down to microliter accuracy
Connecting a Raspberry Pi with an Arduino microcontroller by establishing a serial connection between the two devices, allowing a variety of different tasks
Data of all CADs designed by the TU Darmstadt technical department
A complete construction tutorial including a BOM (Bill of Materials incl. prices)

RESULTS

Circle Detection

The sample detection was facing two problems which on the first glance seem rather simple. Detect a circle and detect a rectangle. The detection of a circle is more easy due to the fact that it is a analytical function. This was already implemented by openCV and we were able to use the circle detection for our sample tracking system. The green circles are the found objects. The red lines are vectors from the central position of our probe to the sample positions. They are used to drive the stepper motors so that our probe can move to them. You can also see, that changing the probes position are recognized and the vectors are recalculated.

Multi-Object Detection

A more delicate task was to detect round samples, stored in a rack. The difficulty is that the objects are intersecting and the algorithm needs to distinguish them somehow. You can see how the camera sees the whole object. Even with our eye it is difficult to see a rectangle. We improved the algorithm step by step using prominent features of the rack. The result is shown in the last picture.

Rectangle Detection

A rectangle detection is more complicated, but we were able to solve this challenge. You can see how the algorithm detect the rectangles and mark them, also tracking the single samples.

Full Functionality

As shown in the video our robot starts with its routine and going from sample to sample, checking for light emission. For demonstration purposes we installed a LED light. One can see, that the optic detect the LED light and then moves down to dispense liquid into the sample, because our pipette is installed sideways on the optical head. Then its moving back to its original position and continues its routine. At the end of the video we show the possible adjustments like brightness and focus.

FURTHER DEVELOPMENTS

Due to a tight time schedule from the start to the end of iGEM it was not possible for us to realize all ideas and planned developments in respect of improvement of the robot itself. For a working process with more kinds of bacteria cultures it is absolutely indispensable to develop a system that is able to avoid all sorts of contamination between the different bacteria. Therefore it would be an option to have an extra reservoir filled with ethanol in which the tip of the syringe can be sterilized between the checks of different samples. Another modification that would be useful for working with individual bacteria cultures is making the power LED's changeable. This is necessary if the the wavelength of the LEDs does not overlap with the absorption spectrum of the fluorescent proteins or overlaps with a part of the spectrum that has a very low absorption efficiency.
Moreover, apart from the latter developments it may be useful to improve the syringe pump system. Instead of using a syringe pump it would be useful to use a system with a reservoir of liquids and a pump that works continously like a turbine, for example see http://www.ardulink.org/automatic-lipid-dispensing/.
Another useful modification of the robot would be to rebuild its foundation, namely an Ultimaker 3D-printer setup. Essential alterations would be to replace the sample stage with a heatbed and to replace the current head with a printhead hotend. Since the current head can be clipped it would not be too much of a challenge. Furthermore, a change of the syringe extruder is necessary, if the printer should work with plastics. An alternative approach is a kind of paste 3D-printer. In this case it wouldn't even be needed to change the head and the syringe, because of the already viscous properties of the paste.

Optics

Operating Range of Wavelengths

The robots optics consists of two big components, a camera head and a lightbox. The camera head is responsible for two tasks, which are the detection and localization of samples, and the fluorescence measurements. The light table illuminates the sample stage uniformly from below, thereby aiding the camera to reliably do the detection work. The fluorescence measurement and object detection are separated in terms of their operating range of wavelengths. All the detection occurs at wavelengths above 860nm, which is near infrared. The light table radiates uniform infrared light, while the camera chip is capable of capturing this wavelength. There is nothing special to the camera chip and in principal all commercially available CCD-Chips can potentially capture near infrared. This is usually an undesirable feature for photographic purposes, since it falsifies image colors. This is why camera lenses are usually equipped with an infrared filter. In our case, we use our own lens system, where we removed the infrared filter. The reason why we chose the detection to operate at infrared is because we are dealing with mVenus, a mutant of eYFP, which does not absorb infrared. Now, if we take a look at the spectra, one can see that the gap between the emission and absorption peaks is pretty small. The absorption peak occurs at wavelength of 512 nm, while the emission peak is located at a wavelength of 528 nm.

Fluorescence Measurement and Filtering

For pulse measurements there is a need for advanced high frequency circuitry, which is capable of forcing the LEDs to emit pulse lengths of picoseconds. LEDs have rise and fall times in the nanosecond region, if they are used in a simple on/off manner. For continuous measurements we have to seperate the wavelength of stimulation from the wavelength of fluorescent emission. Therefore, we used a long pass filter with a cutoff wavelength of 515nm. It is capable of blocking most of the stimulation light, while letting most of the mVenus emission pass. The camera captures a long-exposed image to be further analyzed. To get rid of the residual stimulation light appearing in that image, it is digitally color-filtered, and segmented into regions of interest (ROI). The determination of ROIs occurs simultaneously with the detection of samples. The filtering is very strong, and does indeed block some of the already weak mVenus fluorescence. This is why we use an exposure time of 10 seconds for a fluorescence capture, to make sure enough data is collected.

Optical Hardware - Camera Head

We are using the 8 megapixels PiCamera, because we have access to its capturing settings like framerate, exposure time, gains, light sensitivity etc. over an existing programming interface [1]. This is absolutely necessary since detection and measurement have totally different requirements. Another benefit is, that we are able to capture directly in grayscale (Y part of YUV) for fast detection purposes, and switch to RGB when doing fluorescence measurements. We do not have these degrees of freedom with an ordinary USB camera. However, the disadvantage is, the stock camera itself is equipped with a very minimal lens system. Since our measurements do not only take place at different wavelengths, but also at different distances to the vessels (fluorescence images are taken from each single cuvette, while the camera head is directly placed over its opening), there is a need for adjusting the focus. We have therefore developed a focusing system consisting of a so called voice coil, which inhabits a 3D printed adapter socket for the PiCamera. The adapter socket also includes the optical longpass filter. The voice coil holds a suspended lens, which can be adjusted in its distance to the camera chip. This method is used in most smartphones. In our case, we took our voice coil out of an old webcam. The voice coil is fed with a PWM signal provided directly by the Raspberry Pi's hardware PWM channel. We use a simple L298N H-Bridge stepperdriver to amplify the PWM signal, and to decouple the Raspberry Pi's precious hardware PWM pin. Different duty cycles mean different focal positions. The coil current is tuned with a potentiometer. In this way we are able to automatically focus the lens by evaluating simple Sobel-filter-based sharpness measurements. Our autofocus is finding best sharpness within two seconds, using a robust global search algorithm. It is applied every time a new set of racks and samples is placed into the robot, i.e. prior to each new session. Also, to adjust the focus for individual fluorescence captures, the sharpness of the individual sample corners is considered. The camera head also includes the stimulating LEDs, which are 4 high Power Cree XTE, driven by a 0.9 amp current source and a PWM signal delivered by the Raspberry Pi.
The lightbox is an essential part of the detection. All applied detection algorithms rely on thresholding the image, or filtered versions of it. The thresholding is basically doing a binary selection of relevant versus irrelevant image information. Therefore, there is always a loss of image information. If there is less clutter in the image, then there is no need to use strong thresholds, therefore conserving more of the image information. The lightbox is acting as a clean background, creating only low amounts of static noise and clutter due to its uniform radiation of light, allowing us to use less strict thresholds. It also emphasizes the samples' corners and enhances the detection reliability. The heart of the lightbox is a Plexiglas panel called “Endlighten”. Light is laterally injected and reflected off systematic impurities inside the panel [2]. The light then leaves the panel uniformly in all directions. Light leaving the panel back side is mirrored to the front side by a white reflective Plexiglas, and additionally diffused by a diffusor plate, also made of Plexiglas. The light is injected by flat-end infrared LEDs which are mounted on 3D printed rails and tightly clamped to the sides of the Endlighten panel. The LEDs are driven by constant current sources to give them a long lifetime.

References
[1]: https://picamera.readthedocs.io/en/release-1.12/
[2]: https://www.plexiglas-shop.com/pdfs/en/212-15-PLEXIGLAS-LED-edge-lighting-en.pdf

Cooling

All electronic components produce a significant amount of heat, especially the motor parts, power supply, and the Raspberry Pi. Since the robot chassis is meant to be completely enclosed to keep light out, heat is going to pile up in the upper part of the interior. To protect the samples from temperatures above room temperature, it is necessary to include a cooling system, which ensures proper air circulation, and does not let in ambient light. To fulfill these two requirements we decided to adapt a double-walled cooling system. The simplest implementation of it is based on the fact that warm air rises naturally, and incorporates the power supply as the air intake. The power supply is placed on the bottom of the robot, and draws in fresh air. The air, which is getting warmed up by the interior rises to the top and is let out by a radial fan through an extractor hood, made of a laser cutted MDF grid. The warm air in between the MDF grid and the outer wall is directed through a 3D printed exhaust tunnel. Thus, ambient light is being kept out.

Software

Marlin

Marlin is an Open Source C++ firmware for 3D printers available on GitHub. It is executed on an 8-bit micro-controller, which in our case is an Arduino Mega 2560. Due to the fact, that Marlin is adaptable to a lot of boards and different configurations it was the perfect choice for us to use it in our project. Nevertheless, we had to modify the firmware a little bit, so that it meets all our needs. For example we had to configure so called endstops, that ensure that the sample stage and the head won't crash at a dead end. If you're intereseted in all changes we made, click here.
There, you will also find a link where you can download the original, unmodified version of the marlin firmware.

OpenCV

OpenCV is a cross-platform image processing library and free for use under the open-source BSD license. Its development has been initiated by Intel in the nineties to demonstrate the capability of CPUs in executing complex image processing tasks. OpenCV covers the most basic morphological image operations up to advanced machine learning algorithms. It is written in C++, which is also its primary interface. By now most of the available features have been wrapped for other programming languages like Python, which we are going to use. Especially what we are looking for in OpenCV are its feature extraction algorithms, like the Hough transformation (1) (show details), contour and edge detection (2) (show details), and image moment (3) (show details) extraction. Also we make our lives easier by utilizing its implemented and optimized morphological operators and image filters (4) (show details), like the median filter and the sobel operator.
To enhance the long-term reliability of detection we separate the detection procedure into two steps: Since racks are predominantly used to hold the sample vessels, we firstly extract and detect the rectangular shaped racks with contour and thresholding techniques. Several image shape descriptors are used to track possible positional changes. The user is then allowed to manually assign regional attributes (e.g. radii, heights) for each rack. The assignment of local radii helps the circle detection in the second step. Not only does it inhibit false detections, but also allows for a proper multithreaded apportionment of work.

PyQt

Qt is a software tool to develop a GUI (Graphical User Interface). It is available under a commercial and an open-source license. The software is a cross-platform application framework, which means it runs on the most computer systems like Unix or Windows. The underlying programming language is C++ and Qt can use already existing programming languages like Javascript, making it a powerful tool.
The main idea of Qt is to use a system of signals and slots to have an easy framework to connect displayed elements with underlying functions. Also, the reusability of already existing code is enhanced. Every graphical element, for example a button, emits its own signal when it is pressed or used. The signal then can be used to trigger an action, like closing a window. If the signal is not connected to a function nothing will happen, however the signal will be emitted with no consequences. Now it is possible to connect the emitted signal with a desired action, called slot, and the program gets its specific behavior.
Qt is widely used by companies like the European Space Agency (ESA), Samsung, DreamWorks, Volvo and many more. To be able to combine the possibilities of Qt with the simplicity of the Python programming language, PyQt was developed. PyQt is a binding for Python capable of translating Qt methods within the Python syntax.
To be able to get a direct preview of the constructed GUI, Qt Designer is a helpful tool. Basically it enables an intuitive way to build a graphical user interface without a need to explicitly coding it. To later work with the code itself, PyQt uses a method called pyuic(number), which is executed through the terminal. The number in the brackets stands for the version.
After converting the code one can open the GUI as a regular python script and work with it as usual.

References:
https://riverbankcomputing.com/software/pyqt/intro
https://www.qt.io/
https://en.wikipedia.org/wiki/Qt_(software)

BUILDING INSTRUCTIONS

The following video shows the assembly of our robot's mechanics. Feel free to jump in the video or download it; so you can watch as many times as you like. All necessary steps are commented.

Construction Video

Circuit Diagram

The stepper motors for X and Y are plugged into the RAMPS board as shown in the graphic. There are already prepared cables available with corresponding connectors. To control and power them the stepper driver (A4988) are plugged in the RAMPS slots, marked with X and Y. On the bottom of the stepper driver it is marked which pin is ground and so on. Also on the RAMPS board it is marked where the corresponding slots for ground etc. are. The X and Y stepper motors have one endstop each to provide a stop signal for the homing routine. It is also possible for safety reasons to add one more endstop to each axis, so that the movement stops for sure at the end positons. All the endstops are plugged into the marked position on the RAMPS board. From left to right the pins are for X min and X max position, Y min and Y max position and Z min and Z max position. The pin marked with a + is reserved for the 5 V, the middle pin is the ground and the last pin marked with a S is the signal. The Z stepper motor has two endstops and is connected like the X and Y stepper motors and endstops. To drive our syringe pump one more stepper motor and one endstop is required and is plugged into the E0 position. It is connected like X,Y and Z.
To power the RAMPS board 12 V are needed, for example from an ATX power supply. The Raspberry Pi3 is responsible for controlling the optics and LEDs whereas the Arduino controls the stepper motors and endstops. Arduino and Raspberry Pi are connected via one of the Pi’s USB port and the USB Type B input of the Arduino. The lightbox is powered with 24 V from the ATX power supply. The lightbox consists of 4 LED bars, each carrying 10 infrared LEDs in series. Each bar is driven by one 50mA constant current source. These are parallel connected to the -12V of the power supply, and to the NC of one Relay. The COM port of that Relais is then connected to the +12V of the power supply. The 2 Relay Modul serves as a Raspberry Pi controlled switch which controls the lightbox and the blue exposing LEDs. The corresponding wire’s number refer to the used GPIO pins. Be careful, the internal GPIO layout is different as the numbering on the Pi layout scheme. All numbers correspond to the latter!
The L298N are DC motor drivers which we alienated as protection circuit, to separate the 12 V from the Raspberry Pi’s GPIO. The blue exposing LEDs are series-connected, and controlled by the 2 Relay Modul on the first side.

Required Software

In case you want to rebuild ou project, you can download all needed software in this section. The software includes the exact marlin firmware, which we uploaded on the Arduino, as well as the software that is installed on the Raspberry Pi and some improved python scripts. Furthermore, to enable wireless support, install and configure "hostapd" and "dnsmasq" on your Raspberry Pi.

Bill Of Materials

The costs on this bill are just an example for what we paid for the items. It may be that the costs vary from ours, if you are buying them from another shop as we did. Also, it should be mentioned, that a lot of the items we used in this project are 3D-printed or were manufactured in a workshop, so that we didn't have to pay for it. So it is plausible, that the total costs for this project will rise. Nevertheless, if you are still interested in constructing this robot, you can take a look at the bill of materials here.

Overview
Functionality
Results
Further Developments
Technical Details
Software

Team:TU Darmstadt/Software